Methods and apparatus for evaluating developmental conditions and providing control over coverage and reliability

ABSTRACT

The methods and apparatus disclosed herein can evaluate a subject for a developmental condition or conditions and provide improved sensitivity and specificity for categorical determinations indicating the presence or absence of the developmental condition by isolating hard-to-screen cases as inconclusive. The methods and apparatus disclosed herein can be configured to be tunable to control the tradeoff between coverage and reliability and to adapt to different application settings and can further be specialized to handle different population groups.

CROSS-REFERENCE

The present application is a bypass continuation of International PatentApplication No. PCT/US2017/061552, filed Nov. 14, 2017, which claimspriority to U.S. Provisional Application No. 62/421,958, filed on Nov.14, 2016, and U.S. Provisional Application No. 62/452,908, filed on Jan.31, 2017, each of which application are herein incorporated in theirentireties for all purposes.

The subject matter of the present application is related to U.S.application Ser. No. 14/354,032, filed on Apr. 24, 2014, now U.S. Pat.No. 9,443,205, and U.S. application Ser. No. 15/234,814, filed on Aug.11, 2016, the entire disclosures of which are incorporated herein byreference.

BACKGROUND OF THE INVENTION

Prior methods and apparatus for diagnosing and treating cognitivefunction attributes of people such as, for example, people with adevelopmental condition or disorder can be less than ideal in at leastsome respects. Unfortunately, a less than ideal amount of time, energyand money can be required to obtain a diagnosis and treatment, and todetermine whether a subject is at risk for decreased cognitive functionsuch as, dementia, Alzheimer's or a developmental disorder. Examples ofcognitive and developmental disorders less than ideally treated by theprior approaches include autism, autistic spectrum, attention deficitdisorder, attention deficit hyperactive disorder and speech and learningdisability, for example. Examples of mood and mental illness disordersless than ideally treated by the prior approaches include depression,anxiety, ADHD, obsessive compulsive disorder, and substance disorderssuch as substance abuse and eating disorders. The prior approaches todiagnosis and treatment of several neurodegenerative diseases can beless than ideal in many instances, and examples of suchneurodegenerative diseases include age related cognitive decline,cognitive impairment, Alzheimer's disease, Parkinson's disease,Huntington's disease, and amyotrophic lateral sclerosis (“ALS”), forexample. The healthcare system is under increasing pressure to delivercare at lower costs, and prior methods and apparatus for clinicallydiagnosing or identifying a subject as at risk of a developmentaldisorder can result in greater expense and burden on the health caresystem than would be ideal. Further, at least some subjects are nottreated as soon as ideally would occur, such that the burden on thehealthcare system is increased with the additional care required forthese subjects.

The identification and treatment of cognitive function attributes,including for example, developmental disorders in subjects can present adaunting technical problem in terms of both accuracy and efficiency.Many known methods for identifying such attributes or disorders areoften time-consuming and resource-intensive, requiring a subject toanswer a large number of questions or undergo extensive observationunder the administration of qualified clinicians, who may be limited innumber and availability depending on the subject's geographicallocation. In addition, many known methods for identifying and treatingbehavioral, neurological, or mental health conditions or disorders haveless than ideal accuracy and consistency, as subjects to be evaluatedusing such methods often present a vast range of variation that can bedifficult to capture and classify. A technical solution to such atechnical problem would be desirable, wherein the technical solution canimprove both the accuracy and efficiency of existing methods. Ideally,such a technical solution would reduce the required time and resourcesfor administering a method for identifying and treating attributes ofcognitive function, such as behavioral, neurological or mental healthconditions or disorders, and improve the accuracy and consistency of theidentification outcomes across subjects.

Although prior lengthy tests with questions can be administered tocaretakers such as parents in order to diagnose or identify a subject asat risk for a developmental condition or disorder, such tests can bequite long and burdensome. For example at least some of these tests haveover one hundred questions, and more than one such lengthy test may beadministered further increasing the burden on health care providers andcaretakers. Additional data may be required such as clinical observationof the subject, and clinical visits may further increase the amount oftime and burden on the healthcare system. Consequently, the time betweena subject being identified as needing to be evaluated and beingclinically identified as at risk or diagnosed with a developmental delaycan be several months, and in some instances over a year.

The delay between identified need for an evaluation and clinicaldiagnosis can result in less than ideal care in at least some instances.Some developmental disorders can be treated with timely intervention.However, the large gap between a caretaker initially identifying aprospective as needing an evaluation and clinically diagnosing thesubject or clinically identifying the subject as at risk can result inless than ideal treatment. In at least some instances, a developmentaldisorder may have a treatment window, and the treatment window may bemissed or the subject treated for only a portion of the treatmentwindow.

Although prior methods and apparatus have been proposed to decrease thenumber of questions asked, such prior methods and apparatus can be lessthan ideal in at least some respects. Although prior methods andapparatus have relied on training and test datasets to train andvalidate, respectively, the methods and apparatus, the actual clinicalresults of such methods and apparatus can be less than ideal, as theclinical environment can present more challenging cases than thetraining and test dataset. The clinical environment can present subjectswho may have one or more of several possible developmental disorders,and relying on a subset of questions may result in less than idealsensitivity and specificity of the tested developmental disorder. Also,the use of only one test to diagnose only one developmental disorder,e.g. autism, may provide less than ideal results for diagnosing theintended developmental disorder and other disorders, as subject behaviorfrom other developmental disorders may present confounding variablesthat decrease the sensitivity and specificity of the subset of questionstargeting the one developmental disorder. Also, reliance on apredetermined subset can result in less than ideal results as morequestions than would be ideal may be asked, and the questions asked maynot be the ideal subset of questions for a particular subject.

Further, many subjects may have two or more related disorders orconditions. If each test is designed to diagnose or identify only asingle disorder or condition, a subject presenting with multipledisorders may be required to take multiple tests. The evaluation of asubject using multiple diagnostic tests may be lengthy, expensive,inconvenient, and logistically challenging to arrange. It would bedesirable to provide a way to test a subject using a single diagnostictest that is capable of identifying or diagnosing multiple relateddisorders or conditions with sufficient sensitivity and specificity.

Additionally, it would be helpful if diagnostic methods and treatmentscould be applied to subjects to advance cognitive function for subjectswith advanced, normal and decreased cognitive function. In light of theabove, improved methods and systems of diagnosing and identifyingsubjects at risk for a particular cognitive function attribute such as adevelopmental disorder and for providing improved digital therapeuticsare needed. Ideally such methods and apparatus would require fewerquestions, decreased amounts of time, determine a plurality of cognitivefunction attributes, such as behavioral, neurological or mental healthconditions or disorders, and provide clinically acceptable sensitivityand specificity in a clinical or nonclinical environment, which can beused to monitor and adapt treatment efficacy. Moreover, improved digitaltherapeutics can provide a customized treatment plan for a patient,receive updated diagnostic data in response to the customized treatmentplan to determine progress, and update the treatment plan accordingly.Ideally, such methods and apparatus can also be used to determine thedevelopmental progress of a subject, and offer treatment to advancedevelopmental progress.

SUMMARY OF THE INVENTION

The methods and apparatus disclosed herein can determine a cognitivefunction attribute such as the developmental progress of a subject in aclinical or nonclinical environment. For example, the described methodsand apparatus can identify a subject as developmentally advanced in oneor more areas of development, or identify a subject as developmentallydelayed or at risk of having one or more developmental disorders. Themethods and apparatus disclosed can determine the subject'sdevelopmental progress by evaluating a plurality of characteristics orfeatures of the subject based on an assessment model, wherein theassessment model can be generated from large datasets of relevantsubject populations using machine-learning approaches. The methods andapparatus disclosed herein comprise improved logical structures andprocesses to diagnose a subject with a disorder among a plurality ofdisorders, using a single test.

The methods and apparatus disclosed herein can diagnose or identify asubject as at risk of having one or more cognitive function attributessuch as for example, a subject at risk of having one or moredevelopmental disorders among a plurality of developmental disorders ina clinical or nonclinical setting, with fewer questions, in a decreasedamounts of time, and with clinically acceptable sensitivity andspecificity in a clinical environment. A processor can be configuredwith instructions to identify a most predictive next question, such thata person can be diagnosed or identified as at risk with fewer questions.Identifying the most predictive next question in response to a pluralityof answers has the advantage of increasing the sensitivity and thespecificity with fewer questions. The methods and apparatus disclosedherein can be configured to evaluate a subject for a plurality ofrelated developmental disorders using a single test, and diagnose ordetermine the subject as at risk of one or more of the plurality ofdevelopmental disorders using the single test. Decreasing the number ofquestions presented can be particularly helpful where a subject presentswith a plurality of possible developmental disorders. Evaluating thesubject for the plurality of possible disorders using just a single testcan greatly reduce the length and cost of the evaluation procedure. Themethods and apparatus disclosed herein can diagnose or identify thesubject as at risk for having a single developmental disorder among aplurality of possible developmental disorders that may have overlappingsymptoms.

While the most predictive next question can be determined in many ways,in many instances the most predictive next question is determined inresponse to a plurality of answers to preceding questions that maycomprise prior most predictive next questions. The most predictive nextquestion can be determined statistically, and a set of possible mostpredictive next questions evaluated to determine the most predictivenext question. In many instances, answers to each of the possible mostpredictive next questions are related to the relevance of the question,and the relevance of the question can be determined in response to thecombined feature importance of each possible answer to a question.

The methods and apparatus disclosed herein can categorize a subject intoone of three categories: having one or more developmental conditions,being developmentally normal or typical, or inconclusive (i.e. requiringadditional evaluation to determine whether the subject has anydevelopmental conditions). A developmental condition can be adevelopmental disorder or a developmental advancement. Note that themethods and apparatus disclosed herein are not limited to developmentalconditions, and may be applied to other cognitive function attributes,such as behavioral, neurological or mental health conditions. Themethods and apparatus may initially categorize a subject into one of thethree categories, and subsequently continue with the evaluation of asubject initially categorized as “inconclusive” by collecting additionalinformation from the subject. Such continued evaluation of a subjectinitially categorized as “inconclusive” may be performed continuouslywith a single screening procedure (e.g., containing various assessmentmodules). Alternatively or additionally, a subject identified asbelonging to the inconclusive group may be evaluated using separate,additional screening procedures and/or referred to a clinician forfurther evaluation.

The methods and apparatus disclosed herein can evaluate a subject usinga combination of questionnaires and video inputs, wherein the two inputsmay be integrated mathematically to optimize the sensitivity and/orspecificity of classification or diagnosis of the subject. Optionally,the methods and apparatus can be optimized for different settings (e.g.,primary vs secondary care) to account for differences in expectedprevalence rates depending on the application setting.

The methods and apparatus disclosed herein can account for differentsubject-specific dimensions such as, for example, a subject's age, ageographic location associated with a subject, a subject's gender or anyother subject-specific or demographic data associated with a subject. Inparticular, the methods and apparatus disclosed herein can takedifferent subject-specific dimensions into account in identifying thesubject as at risk of having one or more cognitive function attributessuch as developmental conditions, in order to increase the sensitivityand specificity of evaluation, diagnosis, or classification of thesubject. For example, subjects belonging to different age groups may beevaluated using different machine learning assessment models, each ofwhich can be specifically tuned to identify the one or moredevelopmental conditions in subjects of a particular age group. Each agegroup-specific assessment model may contain a unique group of assessmentitems (e.g., questions, video observations), wherein some of theassessment items may overlap with those of other age groups' specificassessment models.

In addition, the digital personalized medicine systems and methodsdescribed herein can provide digital diagnostics and digitaltherapeutics to patients. The digital personalized medicine system canuse digital data to assess or diagnose symptoms of a patient in waysthat inform personalized or more appropriate therapeutic interventionsand improved diagnoses.

In one aspect, the digital personalized medicine system can comprisedigital devices with processors and associated software that can beconfigured to: use data to assess and diagnose a patient; captureinteraction and feedback data that identify relative levels of efficacy,compliance and response resulting from the therapeutic interventions;and perform data analysis. Such data analysis can include artificialintelligence, including for example machine learning, and/or statisticalmodels to assess user data and user profiles to further personalize,improve or assess efficacy of the therapeutic interventions.

In some instances, the system can be configured to use digitaldiagnostics and digital therapeutics. Digital diagnostics and digitaltherapeutics can comprise a system or methods for digitally collectinginformation and processing and evaluating the provided data to improvethe medical, psychological, or physiological state of an individual. Adigital therapeutic system can apply software based learning to evaluateuser data, monitor and improve the diagnoses and therapeuticinterventions provided by the system.

Digital diagnostics data in the system can comprise data and meta-datacollected from the patient, or a caregiver, or a party that isindependent of the assessed individual. In some instances, the collecteddata can comprise monitoring behaviors, observations, judgments, orassessments made by a party other than the individual. In furtherinstances, the assessment can comprise an adult performing an assessmentor provide data for an assessment of a child or juvenile. The data andmeta-data can be either actively or passively collected in digitalformat via one or more digital devices such as mobile phones, videocapture, audio capture, activity monitors, or wearable digital monitors.

The digital diagnostic uses the data collected by the system about thepatient, which can include complimentary diagnostic data capturedoutside the digital diagnostic, with analysis from tools such as machinelearning, artificial intelligence, and statistical modeling to assess ordiagnose the patient's condition. The digital diagnostic can alsoprovide an assessment of a patient's change in state or performance,directly or indirectly via data and meta-data that can be analyzed andevaluated by tools such as machine learning, artificial intelligence,and statistical modeling to provide feedback into the system to improveor refine the diagnoses and potential therapeutic interventions.

Data assessment and machine learning from the digital diagnostic andcorresponding responses, or lack thereof, from the therapeuticinterventions can lead to the identification of novel diagnoses forpatients and novel therapeutic regimens for both patents and caregivers.

Types of data collected and utilized by the system can include patientand caregiver video, audio, responses to questions or activities, andactive or passive data streams from user interaction with activities,games or software features of the system, for example. Such data canalso include meta-data from patient or caregiver interaction with thesystem, for example, when performing recommended activities. Specificmeta-data examples include data from a user's interaction with thesystem's device or mobile app that captures aspects of the user'sbehaviors, profile, activities, interactions with the software system,interactions with games, frequency of use, session time, options orfeatures selected, and content and activity preferences. Data can alsoinclude data and meta-data from various third party devices such asactivity monitors, games or interactive content.

Digital therapeutics can comprise instructions, feedback, activities orinteractions provided to the patient or caregiver by the system.Examples include suggested behaviors, activities, games or interactivesessions with system software and/or third party devices.

In further aspects, the digital therapeutics methods and systemsdisclosed herein can diagnose and treat a subject at risk of having oneor more behavioral, neurological or mental health conditions ordisorders among a plurality of behavioral, neurological or mental healthconditions or disorders in a clinical or nonclinical setting. Thisdiagnosis and treatment can be accomplished using the methods andsystems disclosed herein with fewer questions, in a decreased amount oftime, and with clinically acceptable sensitivity and specificity in aclinical environment, and can provide treatment recommendations. Thiscan be helpful when a subject initiates treatment based on an incorrectdiagnosis, for example. A processor can be configured with instructionsto identify a most predictive next question or most instructive nextsymptom or observation such that a person can be diagnosed or identifiedas at risk reliably using only the optimal number of questions orobservations. Identifying the most predictive next question or mostinstructive next symptom or observation in response to a plurality ofanswers has the advantage of providing treatment with fewer questionswithout degrading the sensitivity or specificity of the diagnosticprocess. In some instances, an additional processor can be provided topredict or collect information on the next more relevant symptom. Themethods and apparatus disclosed herein can be configured to evaluate andtreat a subject for a plurality of related disorders using a singleadaptive test, and diagnose or determine the subject as at risk of oneor more of the plurality of disorders using the single test. Decreasingthe number of questions presented or symptoms or measurements used canbe particularly helpful where a subject presents with a plurality ofpossible disorders that can be treated. Evaluating the subject for theplurality of possible disorders using just a single adaptive test cangreatly reduce the length and cost of the evaluation procedure andimprove treatment. The methods and apparatus disclosed herein candiagnose and treat subject at risk for having a single disorder among aplurality of possible disorders that may have overlapping symptoms.

The most predictive next question, most instructive next symptom orobservation used for the digital therapeutic treatment can be determinedin many ways. In many instances, the most predictive next question,symptom, or observation can be determined in response to a plurality ofanswers to preceding questions or observation that may comprise priormost predictive next question, symptom, or observation to evaluate thetreatment and provide a closed-loop assessment of the subject. The mostpredictive next question, symptom, or observation can be determinedstatistically, and a set of candidates can be evaluated to determine themost predictive next question, symptom, or observation. In manyinstances, observations or answers to each of the candidates are relatedto the relevance of the question or observation, and the relevance ofthe question or observation can be determined in response to thecombined feature importance of each possible answer to a question orobservation. Once a treatment has been initiated, the questions,symptoms, or observations can be repeated or different questions,symptoms, or observations can be used to more accurately monitorprogress and suggest changes to the digital treatment. The relevance ofa next question, symptom or observation can also depend on the varianceof the ultimate assessment among different answer choices of thequestion or potential options for an observation. For example, aquestion for which the answer choices might have a significant impact onthe ultimate assessment down the line can be deemed more relevant than aquestion for which the answer choices might only help to discerndifferences in severity for one particular condition, or are otherwiseless consequential.

In one aspect, a method of providing an evaluation of at least onecognitive function attribute of a subject may comprise: on a computersystem having a processor and a memory storing a computer program forexecution by the processor, the computer program comprising instructionsfor: receiving data of the subject related to the cognitive functionattribute; evaluating the data of the subject using a machine learningmodel; and providing an evaluation for the subject, the evaluationselected from the group consisting of an inconclusive determination anda categorical determination in response to the data. The machinelearning model may comprise a selected subset of a plurality of machinelearning assessment models.

The categorical determination may comprise a presence of the cognitivefunction attribute and an absence of the cognitive function attribute.Receiving data from the subject may comprise receiving an initial set ofdata. Evaluating the data from the subject may comprise evaluating theinitial set of data using a preliminary subset of tunable machinelearning assessment models selected from the plurality of tunablemachine learning assessment models to output a numerical score for eachof the preliminary subset of tunable machine learning assessment models.

The method may further comprise providing a categorical determination oran inconclusive determination as to the presence or absence of thecognitive function attribute in the subject based on the analysis of theinitial set of data, wherein the ratio of inconclusive to categoricaldeterminations can be adjusted. The method may further comprise:determining whether to apply additional assessment models selected fromthe plurality of tunable machine learning assessment models if theanalysis of the initial set of data yields an inconclusivedetermination; receiving an additional set of data from the subjectbased on an outcome of the decision; evaluating the additional set ofdata from the subject using the additional assessment models to output anumerical score for each of the additional assessment models based onthe outcome of the decision; and providing a categorical determinationor an inconclusive determination as to the presence or absence of thecognitive function attribute in the subject based on the analysis of theadditional set of data from the subject using the additional assessmentmodels, wherein the ratio of inconclusive to categorical determinationscan be adjusted.

The method may further comprise: combining the numerical scores for eachof the preliminary subset of assessment models to generate a combinedpreliminary output score; and mapping the combined preliminary outputscore to a categorical determination or to an inconclusive determinationas to the presence or absence of the cognitive function attribute in thesubject, wherein the ratio of inconclusive to categorical determinationscan be adjusted.

The method may further comprise employing rule-based logic orcombinatorial techniques for combining the numerical scores for each ofthe preliminary subset of assessment models and for combining thenumerical scores for each of the additional assessment models. The ratioof inconclusive to categorical determinations may be adjusted byspecifying an inclusion rate. The categorical determination as to thepresence or absence of the developmental condition in the subject may beassessed by providing a sensitivity and specificity metric. Theinclusion rate may be no less than 70% and the categorical determinationmay result in a sensitivity of at least 70 with a correspondingspecificity of at least 70. The inclusion rate may be no less than 70%and the categorical determination may result in a sensitivity of atleast 80 with a corresponding specificity of at least 80. The inclusionrate may be no less than 70% and the categorical determination mayresult in a sensitivity of at least 90 with a corresponding specificityof at least 90.

Data from the subject may comprise at least one of a sample of adiagnostic instrument, wherein the diagnostic instrument comprises a setof diagnostic questions and corresponding selectable answers, anddemographic data.

The method may further comprise: training a plurality of tunable machinelearning assessment models using data from a plurality of subjectspreviously evaluated for the developmental condition, wherein trainingcomprises: pre-processing the data from the plurality of subjects usingmachine learning techniques; extracting and encoding machine learningfeatures from the pre-processed data; processing the data from theplurality of subjects to mirror an expected prevalence of a cognitivefunction attribute among subjects in an intended application setting;selecting a subset of the processed machine learning features;evaluating each model in the plurality of tunable machine learningassessment models for performance, wherein each model is evaluated forsensitivity and specificity for a pre-determined inclusion rate; anddetermining an optimal set of parameters for each model based ondetermining the benefit of using all models in a selected subset of theplurality of tunable machine learning assessment models. Determining anoptimal set of parameters for each model may comprise tuning theparameters of each model under different tuning parameter settings.

Processing the encoded machine learning features may comprise: computingand assigning sample weights to every sample of data, wherein eachsample of data corresponds to a subject in the plurality of subjects,wherein samples are grouped according to subject-specific dimensions,and wherein the sample weights are computed and assigned to balance onegroup of samples against every other group of samples to mirror theexpected distribution of each dimension among subjects in an intendedsetting. The subject-specific dimensions may comprise a subject'sgender, the geographic region where a subject resides, and a subject'sage. Extracting and encoding machine learning features from thepre-processed data may comprise using feature encoding techniques suchas but not limited to one-hot encoding, severity encoding, andpresence-of-behavior encoding. Selecting a subset of the processedmachine learning features may comprise using bootstrapping techniques toidentify a subset of discriminating features from the processed machinelearning features.

The cognitive function attribute may comprise a behavioral disorder anda developmental advancement. The categorical determination provided forthe subject may be selected from the group consisting of an inconclusivedetermination, a presence of multiple cognitive function attributes, andan absence of multiple cognitive function attributes in response to thedata.

In another aspect, an apparatus to evaluate a cognitive functionattribute of a subject may comprise processor configured withinstructions that, when executed, cause the processor to perform themethod described above.

In another aspect, a mobile device for providing an evaluation of atleast one cognitive function attribute of a subject may comprise: adisplay; and a processor configured with instructions to: receive anddisplay data of the subject related to the cognitive function attribute;and receive and display an evaluation for the subject, the evaluationselected from the group consisting of an inconclusive determination anda categorical determination; wherein the evaluation for the subject hasbeen determined in response to the data of the subject.

The categorical determination may be selected from the group consistingof a presence of the cognitive function attribute, and an absence of thecognitive function attribute. The cognitive function attribute may bedetermined with a sensitivity of at least 80 and a specificity of atleast 80, respectively, for the presence or the absence of the cognitivefunction attribute. The cognitive function attribute may be determinedwith a sensitivity of at least 90 and a specificity of at least 90,respectively, for the presence or the absence of the cognitive functionattribute. The cognitive function attribute may comprise a behavioraldisorder and a developmental advancement.

In another aspect, digital therapeutic system to treat a subject with apersonal therapeutic treatment plan may comprise: one or more processorscomprising software instructions; a diagnostic module to receive datafrom the subject and output diagnostic data for the subject, thediagnostic module comprising one or more classifiers built using machinelearning or statistical modeling based on a subject population todetermine the diagnostic data for the subject, and wherein thediagnostic data comprises an evaluation for the subject, the evaluationselected from the group consisting of an inconclusive determination anda categorical determination in response to data received from thesubject; and a therapeutic module to receive the diagnostic data andoutput the personal therapeutic treatment plan for the subject, thetherapeutic module comprising one or more models built using machinelearning or statistical modeling based on at least a portion the subjectpopulation to determine and output the personal therapeutic treatmentplan of the subject, wherein the diagnostic module is configured toreceive updated subject data from the subject in response to therapy ofthe subject and generate updated diagnostic data from the subject andwherein the therapeutic module is configured to receive the updateddiagnostic data and output an updated personal treatment plan for thesubject in response to the diagnostic data and the updated diagnosticdata.

The diagnostic module may comprise a diagnostic machine learningclassifier trained on the subject population and the therapeutic modulemay comprise a therapeutic machine learning classifier trained on the atleast the portion of the subject population and the diagnostic moduleand the therapeutic module may be arranged for the diagnostic module toprovide feedback to the therapeutic module based on performance of thetreatment plan. The therapeutic classifier may comprise instructionstrained on a data set comprising a population of which the subject isnot a member and the subject may comprise a person who is not a memberof the population. The diagnostic module may comprise a diagnosticclassifier trained on plurality of profiles of a subject population ofat least 10,000 people and therapeutic profile trained on the pluralityof profiles of the subject population.

In another aspect, a system to evaluate of at least one cognitivefunction attribute of a subject may comprise: a processor configuredwith instructions that when executed cause the processor to: present aplurality of questions from a plurality of chains of classifiers, theplurality of chains of classifiers comprising a first chain comprising asocial/behavioral delay classifier and a second chain comprising aspeech & language delay classifier. The social/behavioral delayclassifier may be operatively coupled to an autism & ADHD classifier.The social/behavioral delay classifier may be configured to output apositive result if the subject has a social/behavioral delay and anegative result if the subject does not have the social/behavioraldelay. The social/behavioral delay classifier may be configured tooutput an inconclusive result if it cannot be determined with aspecified sensitivity and specificity whether or not the subject has thesocial/behavioral delay. The social/behavioral delay classifier outputmay be coupled to an input of an Autism and ADHD classifier and theAutism and ADHD classifier may be configured to output a positive resultif the subject has Autism or ADHD. The output of the Autism and ADHDclassifier may be coupled to an input of an Autism v. ADHD classifier,and the Autism v. ADHD classifier may be configured to generate a firstoutput if the subject has autism and a second output if the subject hasADHD. The Autism v. ADHD classifier may be configured to provide aninconclusive output if it cannot be determined with specifiedsensitivity and specificity whether or not the subject has autism orADHD. The speech & language delay classifier may be operatively coupledto an intellectual disability classifier. The speech & language delayclassifier may be configured to output a positive result if the subjecthas a speech and language delay and a negative output if the subjectdoes not have the speech and language delay. The speech & language delayclassifier may be configured to output an inconclusive result if itcannot be determined with a specified sensitivity and specificitywhether or not the subject has the speech and language delay. The speech& language delay classifier output may be coupled to an input of anintellectual disability classifier and the intellectual disabilityclassifier may be configured to generate a first output if the subjecthas intellectual disability and a second output if the subject has thespeech and language delay but no intellectual disability. Theintellectual disability classifier may be configured to provide aninconclusive output if it cannot be determined with a specifiedsensitivity and specificity whether or not the subject has theintellectual disability.

The processor may be configured with instructions to present questionsfor each chain in sequence and skip overlapping questions. The firstchain may comprise the social/behavioral delay classifier coupled to anautism & ADHD classifier. The second chain may comprise the speech &language delay classifier coupled to an intellectual disabilityclassifier. A user may go through the first chain and the second chainin sequence.

In another aspect, a method for administering a drug to a subject maycomprise: detecting a neurological disorder of the subject with amachine learning classifier; and administering the drug to the subjectin response to the detected neurological disorder. The neurologicaldisorder may comprise autism spectrum disorder, and the drug may beselected from the group consisting of risperidone, quetiapine,amphetamine, dextroamphetamine, methylphenidate, methamphetamine,dextroamphetamine, dexmethylphenidate, guanfacine, atomoxetine,lisdexamfetamine, clonidine, and aripiprazolecomprise; or theneurological disorder may comprise attention deficit disorder (ADD), andthe drug may be selected from the group consisting of amphetamine,dextroamphetamine, methylphenidate, methamphetamine, dextroamphetamine,dexmethylphenidate, guanfacine, atomoxetine, lisdexamfetamine,clonidine, and modafinil; or the neurological disorder may compriseattention deficit hyperactivity disorder (ADHD), and the drug may beselected from the group consisting of amphetamine, dextroamphetamine,methylphenidate, methamphetamine, dextroamphetamine, dexmethylphenidate,guanfacine, atomoxetine, lisdexamfetamine, clonidine, and modafinil; orthe neurological disorder may comprise obsessive-compulsive disorder,and the drug may be selected from the group consisting of buspirone,sertraline, escitalopram, citalopram, fluoxetine, paroxetine,venlafaxine, clomipramine, and fluvoxamine; or the neurological disordermay comprise acute stress disorder, and the drug may be selected fromthe group consisting of propranolol, citalopram, escitalopram,sertraline, paroxetine, fluoextine, venlafaxine, mirtazapine,nefazodone, carbamazepine, divalproex, lamotrigine, topiramate,prazosin, phenelzine, imipramine, diazepam, clonazepam, lorazepam, andalprazolam; or the neurological disorder may comprise adjustmentdisorder, and the drug may be selected from the group consisting ofbusiprone, escitalopram, sertraline, paroxetine, fluoextine, diazepam,clonazepam, lorazepam, and alprazolam; or neurological disorder maycomprise agoraphobia, and the drug may be selected from the groupconsisting of diazepam, clonazepam, lorazepam, alprazolam, citalopram,escitalopram, sertraline, paroxetine, fluoextine, and busiprone; or theneurological disorder may comprise Alzheimer's disease, and the drug maybe selected from the group consisting of donepezil, galantamine,memantine, and rivastigmine; or the neurological disorder may compriseanorexia nervosa, and the drug may be selected from the group consistingof olanzapine, citalopram, escitalopram, sertraline, paroxetine, andfluoxetine; or the neurological disorder may comprise anxiety disorders,and the drug may be selected from the group consisting of sertraline,escitalopram, citalopram, fluoxetine, diazepam, buspirone, venlafaxine,duloxetine, imipramine, desipramine, clomipramine, lorazepam,clonazepam, and pregabalin; or the neurological disorder may comprisebereavement, and the drug may be selected from the group consisting ofcitalopram, duloxetine, and doxepin; or the neurological disorder maycomprise binge eating disorder, and the drug may be selected from thegroup consisting of lisdexamfetamine; or the neurological disorder maycomprise bipolar disorder, and the drug may be selected from the groupconsisting of topiramate, lamotrigine, oxcarbazepine, haloperidol,risperidone, quetiapine, olanzapine, aripiprazole, and fluoxetine; orthe neurological disorder may comprise body dysmorphic disorder, and thedrug may be selected from the group consisting of sertraline,escitalopram, and citalopram; or the neurological disorder may comprisebrief psychotic disorder, and the drug may be selected from the groupconsisting of clozapine, asenapine, olanzapine, and quetiapine; or theneurological disorder may comprise bulimia nervosa, and the drug may beselected from the group consisting of sertraline and fluoxetine; or theneurological disorder may comprise conduct disorder, and the drug may beselected from the group consisting of lorazepam, diazepam, and clobazam;or the neurological disorder may comprise delusional disorder, and thedrug may be selected from the group consisting of clozapine, asenapine,risperidone, venlafaxine, bupropion, and buspirone; the neurologicaldisorder may comprise depersonalization disorder, and the drug may beselected from the group consisting of sertraline, fluoxetine,alprazolam, diazepam, and citalopram; or the neurological disorder maycomprise depression, and the drug may be selected from the groupconsisting of sertraline, fluoxetine, citalopram, bupropion,escitalopram, venlafaxine, aripiprazole, buspirone, vortioxetine, andvilazodone; or the neurological disorder may comprise disruptive mooddysregulation disorder, and the drug may be selected from the groupconsisting of quetiapine, clozapine, asenapine, and pimavanserin; or theneurological disorder may comprise dissociative amnesia, and the drugmay be selected from the group consisting of alprazolam, diazepam,lorazepam, and chlordiazepoxide; or the neurological disorder maycomprise dissociative disorder, and the drug may be selected from thegroup consisting of bupropion, vortioxetine, and vilazodone; or theneurological disorder may comprise dissociative fugue, and the drug maybe selected from the group consisting of amobarbital, aprobarbital,butabarbital, and methohexitlal; or the neurological disorder maycomprise dysthymic disorder, and the drug may be selected from the groupconsisting of bupropion, venlafaxine, sertraline, and citalopram; theneurological disorder may comprise eating disorders, and the drug may beselected from the group consisting of olanzapine, citalopram,escitalopram, sertraline, paroxetine, and fluoxetine; or theneurological disorder may comprise gender dysphoria, and the drug may beselected from the group consisting of estrogen, prostogen, andtestosterone; or the neurological disorder may comprise generalizedanxiety disorder, and the drug may be selected from the group consistingof venlafaxine, duloxetine, buspirone, sertraline, and fluoxetine; orthe neurological disorder may comprise hoarding disorder, and the drugmay be selected from the group consisting of buspirone, sertraline,escitalopram, citalopram, fluoxetine, paroxetine, venlafaxine, andclomipramine; or the neurological disorder may comprise intermittentexplosive disorder, and the drug may be selected from the groupconsisting of asenapine, clozapine, olanzapine, and pimavanserin; or theneurological disorder may comprise kleptomania, and the drug may beselected from the group consisting of escitalopram, fluvoxamine,fluoxetine, and paroxetine; or the neurological disorder may comprisepanic disorder, and the drug may be selected from the group consistingof bupropion, vilazodone, and vortioxetine; or the neurological disordermay comprise Parkinson's disease, and the drug may be selected from thegroup consisting of rivastigmine, selegiline, rasagiline, bromocriptine,amantadine, cabergoline, and benztropine; or the neurological disordermay comprise pathological gambling, and the drug may be selected fromthe group consisting of bupropion, vilazodone, and vartioxetine; or theneurological disorder may comprise postpartum depression, and the drugmay be selected from the group consisting of sertraline, fluoxetine,citalopram, bupropion, escitalopram, venlafaxine, aripiprazole,buspirone, vortioxetine, and vilazodone; or the neurological disordermay comprise posttraumatic stress disorder, and the drug may be selectedfrom the group consisting of sertraline, fluoxetine, and paroxetine; orthe neurological disorder may comprise premenstrual dysphoric disorder,and the drug may be selected from the group consisting of estadiol,drospirenone, sertraline, citalopram, fluoxetine, and busiprone; or theneurological disorder may comprise pseudobulbar affect, and the drug maybe selected from the group consisting of dextromethorphan hydrobromide,and quinidine sulfate; or the neurological disorder may comprisepyromania, and the drug may be selected from the group consisting ofclozapine, asenapine, olanzapine, paliperidone, and quetiapine; or theneurological disorder may comprise schizoaffective disorder, and thedrug may be selected from the group consisting of sertraline,carbamazepine, oxcarbazepine, valproate, haloperidol, olanzapine, andloxapine; or the neurological disorder may comprise schizophrenia, andthe drug may be selected from the group consisting of chlopromazine,haloperidol, fluphenazine, risperidone, quetiapine, ziprasidone,olanzapine, perphenazine, aripiprazole, and prochlorperazine; or theneurological disorder may comprise schizophreniform disorder, and thedrug may be selected from the group consisting of paliperidone,clozapine, and risperidone; or the neurological disorder may compriseseasonal affective disorder, and the drug may be selected from the groupconsisting of sertraline, and fluoxetine; or the neurological disordermay comprise shared psychotic disorder, and the drug may be selectedfrom the group consisting of clozapine, pimavanserin, risperidone, andlurasidone; or the neurological disorder may comprise social anxietyphobia, and the drug may be selected from the group consisting ofamitriptyline, bupropion, citalopram, fluoxetine, sertraline, andvenlafaxine; or the neurological disorder may comprise specific phobia,and the drug may be selected from the group consisting of diazepam,estazolam, quazepam, and alprazolam; or the neurological disorder maycomprise stereotypic movement disorder, and the drug may be selectedfrom the group consisting of risperidone, and clozapine; or theneurological disorder may comprise Tourette's disorder, and the drug maybe selected from the group consisting of haloperidol, fluphenazine,risperidone, ziprasidone, pimozide, perphenazine, and aripiprazole; orthe neurological disorder may comprise transient tic disorder, and thedrug may be selected from the group consisting of guanfacine, clonidine,pimozide, risperidone, citalopram, escitalopram, sertraline, paroxetine,and fluoxetine; or the neurological disorder may comprisetrichotillomania, and the drug may be selected from the group consistingof sertraline, fluoxetine, paroxetine, desipramine, and clomipramine.

Amphetamine may be administered with a dosage of 5 mg to 50 mg.Dextroamphetamine may be administered with a dosage that is in a rangeof 5 mg to 60 mg. Methylphenidate may be administered with a dosage thatis in a range of 5 mg to 60 mg. Methamphetamine may be administered witha dosage that is in a range of 5 mg to 25 mg. Dexmethylphenidate may beadministered with a dosage that is in a range of 2.5 mg to 40 mg.Guanfacine may be administered with a dosage that is in a range of 1 mgto 10 mg. Atomoxetine may be administered with a dosage that is in arange of 10 mg to 100 mg. Lisdexamfetamine may be administered with adosage that is in a range of 30 mg to 70 mg. Clonidine may beadministered with a dosage that is in a range of 0.1 mg to 0.5 mg.Modafinil may be administered with a dosage that is in a range of 100 mgto 500 mg. Risperidone may be administered with a dosage that is in arange of 0.5 mg to 20 mg. Quetiapine may be administered with a dosagethat is in a range of 25 mg to 1000 mg. Buspirone may be administeredwith a dosage that is in a range of 5 mg to 60 mg. Sertraline may beadministered with a dosage of up to 200 mg. Escitalopram may beadministered with a dosage of up to 40 mg. Citalopram may beadministered with a dosage of up to 40 mg. Fluoxetine may beadministered with a dosage that is in a range of 40 mg to 80 mg.Paroxetine may be administered with a dosage that is in a range of 40 mgto 60 mg. Venlafaxine may be administered with a dosage of up to 375 mg.Clomipramine may be administered with a dosage of up to 250 mg.Fluvoxamine may be administered with a dosage of up to 300 mg.

The machine learning classifier may have an inclusion rate of no lessthan 70%. The machine learning classifier may be capable of outputtingan inconclusive result.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIGS. 1A and 1B show some exemplary developmental disorders that may beevaluated using the assessment procedure as described herein.

FIG. 2 is a schematic diagram of an exemplary data processing module forproviding the assessment procedure as described herein.

FIG. 3 is a schematic diagram illustrating a portion of an exemplaryassessment model based on a Random Forest classifier.

FIG. 4 is an exemplary operational flow of a prediction module asdescribed herein.

FIG. 5 is an exemplary operational flow of a feature recommendationmodule as described herein.

FIG. 6 is an exemplary operational flow of an expected featureimportance determination algorithm as performed by a featurerecommendation module described herein.

FIG. 7 illustrates a method of administering an assessment procedure asdescribed herein.

FIG. 8 shows a computer system suitable for incorporation with themethods and apparatus described herein.

FIG. 9 shows receiver operating characteristic (ROC) curves mappingsensitivity versus fall-out for an exemplary assessment model asdescribed herein.

FIG. 10 is a scatter plot illustrating a performance metric for afeature recommendation module as described herein.

FIG. 11 is an exemplary operational flow of an evaluation module asdescribed herein.

FIG. 12 is an exemplary operational flow of a model tuning module asdescribed herein.

FIG. 13 is another exemplary operational flow of an evaluation module asdescribed herein.

FIG. 14 is an exemplary operational flow of the model output combiningstep depicted in FIG. 13.

FIG. 15 shows an exemplary questionnaire screening algorithm configuredto provide only categorical determinations as described herein.

FIG. 16 shows an exemplary questionnaire screening algorithm configuredto provide categorical and inconclusive determinations as describedherein.

FIG. 17 shows a comparison of the performance for various algorithms forall samples as described herein.

FIG. 18 shows a comparison of the performance for various algorithms forsamples taken from Children Under 4 as described herein.

FIG. 19 shows a comparison of the performance for various algorithms forsamples taken from Children 4 and Over described herein.

FIG. 20 shows the specificity across algorithms at 75%-85% sensitivityrange for all samples as described herein.

FIG. 21 shows the specificity across algorithms at 75%-85% sensitivityrange for Children Under 4 as described herein.

FIG. 22 shows the specificity across algorithms at 75%-85% sensitivityrange for Children 4 and Over as described herein.

FIG. 23A illustrates an exemplary system diagram for a digitalpersonalized medicine platform.

FIG. 23B illustrates a detailed diagram of an exemplary diagnosismodule.

FIG. 23C illustrates a diagram of an exemplary therapy module.

FIG. 24 illustrates an exemplary method for diagnosis and therapy to beprovided in a digital personalized medicine platform.

FIG. 25 illustrates an exemplary flow diagram showing the handling ofautism-related developmental delay.

FIG. 26 illustrates an overall of data processing flows for a digitalpersonalized medical system comprising a diagnostic module and atherapeutic module, configured to integrate information from multiplesources.

FIG. 27 shows a system for evaluating a subject for multiple clinicalindications.

FIG. 28 shows a drug that may be administered in response to a diagnosisby the systems and methods described herein.

DETAILED DESCRIPTION OF THE INVENTION

While various embodiments of the invention have been shown and describedherein, it will be obvious to those skilled in the art that suchembodiments are provided by way of example only. Numerous variations,changes, and substitutions may occur to those skilled in the art withoutdeparting from the invention. It should be understood that variousalternatives to the embodiments of the invention described herein may beemployed. It shall be understood that different aspects of the inventioncan be appreciated individually, collectively, or in combination witheach other.

The terms “based on” and “in response to” are used interchangeably withthe present disclosure.

The term “processor” encompasses one or more of a local processor, aremote processor, or a processor system, and combinations thereof.

The term “feature” is used herein to describe a characteristic orattribute that is relevant to determining the developmental progress ofa subject. For example, a “feature” may refer to a clinicalcharacteristic that is relevant to clinical evaluation or diagnosis of asubject for one or more developmental disorders (e.g., age, ability ofsubject to engage in pretend play, etc.). The term “feature value” isherein used to describe a particular subject's value for thecorresponding feature. For example, a “feature value” may refer to aclinical characteristic of a subject that is related to one or moredevelopmental disorders (e.g., if feature is “age”, feature value couldbe 3; if feature is “ability of subject to engage in pretend play”,feature value could be “variety of pretend play” or “no pretend play”).

As used herein, the phrases “autism” and “autism spectrum disorder” maybe used interchangeably.

As used herein, the phrases “attention deficit disorder (ADD)” and“attention deficit/hyperactivity disorder (ADHD)” may be usedinterchangeably.

Described herein are methods and apparatus for determining thedevelopmental progress of a subject. For example, the described methodsand apparatus can identify a subject as developmentally advanced in oneor more areas of development or cognitively declining in one or morecognitive functions, or identify a subject as developmentally delayed orat risk of having one or more developmental disorders. The methods andapparatus disclosed can determine the subject's developmental progressby evaluating a plurality of characteristics or features of the subjectbased on an assessment model, wherein the assessment model can begenerated from large datasets of relevant subject populations usingmachine-learning approaches.

While methods and apparatus are herein described in the context ofidentifying one or more developmental disorders of a subject, themethods and apparatus are well-suited for use in determining anydevelopmental progress of a subject. For example, the methods andapparatus can be used to identify a subject as developmentally advanced,by identifying one or more areas of development in which the subject isadvanced. To identify one or more areas of advanced development, themethods and apparatus may be configured to assess one or more featuresor characteristics of the subject that are related to advanced or giftedbehaviors, for example. The methods and apparatus as described can alsobe used to identify a subject as cognitively declining in one or morecognitive functions, by evaluating the one or more cognitive functionsof the subject.

Described herein are methods and apparatus for diagnosing or assessingrisk for one or more developmental disorders in a subject. The methodmay comprise providing a data processing module, which can be utilizedto construct and administer an assessment procedure for screening asubject for one or more of a plurality of developmental disorders orconditions. The assessment procedure can evaluate a plurality offeatures or characteristics of the subject, wherein each feature can berelated to the likelihood of the subject having at least one of theplurality of developmental disorders screenable by the procedure. Eachfeature may be related to the likelihood of the subject having two ormore related developmental disorders, wherein the two or more relateddisorders may have one or more related symptoms. The features can beassessed in many ways. For example, the features may be assessed via asubject's answers to questions, observations of a subject, or results ofa structured interaction with a subject, as described in further detailherein.

To distinguish among a plurality of developmental disorders of thesubject within a single screening procedure, the procedure candynamically select the features to be evaluated in the subject duringadministration of the procedure, based on the subject's values forpreviously presented features (e.g., answers to previous questions). Theassessment procedure can be administered to a subject or a caretaker ofthe subject with a user interface provided by a computing device. Thecomputing device comprises a processor having instructions storedthereon to allow the user to interact with the data processing modulethrough a user interface. The assessment procedure may take less than 10minutes to administer to the subject, for example 5 minutes or less.Thus, apparatus and methods described herein can provide a prediction ofa subject's risk of having one or more of a plurality of developmentaldisorders using a single, relatively short screening procedure.

The methods and apparatus disclosed herein can be used to determine amost relevant next question related to a feature of a subject, based onpreviously identified features of the subject. For example, the methodsand apparatus can be configured to determine a most relevant nextquestion in response to previously answered questions related to thesubject. A most predictive next question can be identified after eachprior question is answered, and a sequence of most predictive nextquestions and a corresponding sequence of answers generated. Thesequence of answers may comprise an answer profile of the subject, andthe most predictive next question can be generated in response to theanswer profile of the subject.

The methods and apparatus disclosed herein are well suited forcombinations with prior questions that can be used to diagnose oridentify the subject as at risk in response to fewer questions byidentifying the most predictive next question in response to theprevious answers, for example.

In one aspect, a method of providing an evaluation of at least onecognitive function attribute of a subject comprises the operations of:on a computer system having a processor and a memory storing a computerprogram for execution by the processor. The computer program maycomprise instructions for: 1) receiving data of the subject related tothe cognitive function attribute; 2) evaluating the data of the subjectusing a machine learning model; and 3) providing an evaluation for thesubject. The evaluation may be selected from the group consisting of aninconclusive determination and a categorical determination in responseto the data. The machine learning model may comprise a selected subsetof a plurality of machine learning assessment models. The categoricaldetermination may comprise a presence of the cognitive functionattribute and an absence of the cognitive function attribute.

Receiving data from the subject may comprise receiving an initial set ofdata. Evaluating the data from the subject may comprise evaluating theinitial set of data using a preliminary subset of tunable machinelearning assessment models selected from the plurality of tunablemachine learning assessment models to output a numerical score for eachof the preliminary subset of tunable machine learning assessment models.The method may further comprise providing a categorical determination oran inconclusive determination as to the presence or absence of thecognitive function attribute in the subject based on the analysis of theinitial set of data, wherein the ratio of inconclusive to categoricaldeterminations can be adjusted.

The method may further comprise the operations of: 1) determiningwhether to apply additional assessment models selected from theplurality of tunable machine learning assessment models if the analysisof the initial set of data yields an inconclusive determination; 2)receiving an additional set of data from the subject based on an outcomeof the decision; 3) evaluating the additional set of data from thesubject using the additional assessment models to output a numericalscore for each of the additional assessment models based on the outcomeof the decision; and 4) providing a categorical determination or aninconclusive determination as to the presence or absence of thecognitive function attribute in the subject based on the analysis of theadditional set of data from the subject using the additional assessmentmodels. The ratio of inconclusive to categorical determinations may beadjusted.

The method may further comprise the operations: 1) combining thenumerical scores for each of the preliminary subset of assessment modelsto generate a combined preliminary output score; and 2) mapping thecombined preliminary output score to a categorical determination or toan inconclusive determination as to the presence or absence of thecognitive function attribute in the subject. The ratio of inconclusiveto categorical determinations may be adjusted. The method may furthercomprise the operations of: 1) combining the numerical scores for eachof the additional assessment models to generate a combined additionaloutput score; and 2) mapping the combined additional output score to acategorical determination or to an inconclusive determination as to thepresence or absence of the cognitive function attribute in the subject.The ratio of inconclusive to categorical determinations may be adjusted.The method may further comprise employing rule-based logic orcombinatorial techniques for combining the numerical scores for each ofthe preliminary subset of assessment models and for combining thenumerical scores for each of the additional assessment models.

The ratio of inconclusive to categorical determinations may be adjustedby specifying an inclusion rate and wherein the categoricaldetermination as to the presence or absence of the developmentalcondition in the subject is assessed by providing a sensitivity andspecificity metric. The inclusion rate may be no less than 70% with thecategorical determination resulting in a sensitivity of at least 70 witha corresponding specificity in of at least 70. The inclusion rate may beno less than 70% with the categorical determination resulting in asensitivity of at least 80 with a corresponding specificity in of atleast 80. The inclusion rate may be no less than 70% with thecategorical determination resulting in a sensitivity of at least 90 witha corresponding specificity in of at least 90. The data from the subjectmay comprise at least one of a sample of a diagnostic instrument,wherein the diagnostic instrument comprises a set of diagnosticquestions and corresponding selectable answers, and demographic data.

The method may further comprise training a plurality of tunable machinelearning assessment models using data from a plurality of subjectspreviously evaluated for the developmental condition. The training maycomprise the operations of: 1) pre-processing the data from theplurality of subjects using machine learning techniques; 2) extractingand encoding machine learning features from the pre-processed data; 3)processing the data from the plurality of subjects to mirror an expectedprevalence of a cognitive function attribute among subjects in anintended application setting; 4) selecting a subset of the processedmachine learning features; 5) evaluating each model in the plurality oftunable machine learning assessment models for performance; and 6)determining an optimal set of parameters for each model based ondetermining the benefit of using all models in a selected subset of theplurality of tunable machine learning assessment models. Each model maybe evaluated for sensitivity and specificity for a pre-determinedinclusion rate. Determining an optimal set of parameters for each modelmay comprise tuning the parameters of each model under different tuningparameter settings. Processing the encoded machine learning features maycomprise computing and assigning sample weights to every sample of data.Each sample of data may correspond to a subject in the plurality ofsubjects. Samples may be grouped according to subject-specificdimensions. Sample weights may be computed and assigned to balance onegroup of samples against every other group of samples to mirror theexpected distribution of each dimension among subjects in an intendedsetting. The subject-specific dimensions may comprise a subject'sgender, the geographic region where a subject resides, and a subject'sage. Extracting and encoding machine learning features from thepre-processed data may comprise using feature encoding techniques suchas but not limited to one-hot encoding, severity encoding, andpresence-of-behavior encoding. Selecting a subset of the processedmachine learning features may comprise using bootstrapping techniques toidentify a subset of discriminating features from the processed machinelearning features.

The cognitive function attribute may comprise a behavioral disorder anda developmental advancement. The categorical determination provided forthe subject may be selected from the group consisting of an inconclusivedetermination, a presence of multiple cognitive function attributes andan absence of multiple cognitive function attributes in response to thedata.

In another aspect, an apparatus to evaluate a cognitive functionattribute of a subject may comprise a processor. The processor may beconfigured with instructions that, when executed, cause the processor toreceive data of the subject related to the cognitive function attributeand applies rules to generate a categorical determination for thesubject. The categorical determination may be selected from a groupconsisting of an inconclusive determination, a presence of the cognitivefunction attribute, and an absence of the cognitive function attributein response to the data. The cognitive function attribute may bedetermined with a sensitivity of at least 70 and a specificity of atleast 70, respectively, for the presence or the absence of the cognitivefunction attribute. The cognitive function attribute may be selectedfrom a group consisting of autism, autistic spectrum, attention deficitdisorder, attention deficit hyperactive disorder and speech and learningdisability. The cognitive function attribute may be determined with asensitivity of at least 80 and a specificity of at least 80,respectively, for the presence or the absence of the cognitive functionattribute. The cognitive function attribute may be determined with asensitivity of at least 90 and a specificity of at least 90,respectively, for the presence or the absence of the cognitive functionattribute. The cognitive function attribute may comprise a behavioraldisorder and a developmental advancement.

In another aspect, a non-transitory computer-readable storage mediaencoded with a computer program including instructions executable by aprocessor to evaluate a cognitive function attribute of a subjectcomprises a database, recorded on the media. The database may comprisedata of a plurality of subjects related to at least one cognitivefunction attribute and a plurality of tunable machine learningassessment models; an evaluation software module; and a model tuningsoftware module. The evaluation software module may compriseinstructions for: 1) receiving data of the subject related to thecognitive function attribute; 2) evaluating the data of the subjectusing a selected subset of a plurality of machine learning assessmentmodels; and 3) providing a categorical determination for the subject,the categorical determination selected from the group consisting of aninconclusive determination, a presence of the cognitive functionattribute and an absence of the cognitive function attribute in responseto the data. The model tuning software module may comprise instructionsfor: 1) pre-processing the data from the plurality of subjects usingmachine learning techniques; 2) extracting and encoding machine learningfeatures from the pre-processed data; 3) processing the encoded machinelearning features to mirror an expected distribution of subjects in anintended application setting; 4) selecting a subset of the processedmachine learning features; 5) evaluating each model in the plurality oftunable machine learning assessment models for performance; 6) tuningthe parameters of each model under different tuning parameter settings;and 7) determining an optimal set of parameters for each model based ondetermining the benefit of using all models in a selected subset of theplurality of tunable machine learning assessment models. Each model maybe evaluated for sensitivity and specificity for a pre-determinedinclusion rate. The cognitive function attribute may comprise abehavioral disorder and a developmental advancement.

In another aspect, a computer-implemented system may comprise a digitalprocessing device. The digital processing may comprise at least oneprocessor, an operating system configured to perform executableinstructions, a memory, and a computer program. The memory may comprisestorage for housing data of a plurality of subjects related to at leastone cognitive function attribute and storage for housing a plurality ofmachine learning assessment models. The computer program may includeinstructions executable by the digital processing device for: 1)receiving data of the subject related to the cognitive functionattribute; 2) evaluating the data of the subject using a selected subsetof a plurality of machine learning assessment models; and 3) providing acategorical determination for the subject, the categorical determinationselected from the group consisting of an inconclusive determination, apresence of the cognitive function attribute and an absence of thecognitive function attribute in response to the data. The cognitivefunction attribute may comprise a behavioral disorder and adevelopmental advancement.

In another aspect, a mobile device for providing an evaluation of atleast one cognitive function attribute of a subject may comprise adisplay and a processor. The processor may be configured withinstructions to receive and display data of the subject related to thecognitive function attribute and receive and display an evaluation forthe subject. The evaluation may be selected from the group consisting ofan inconclusive determination and a categorical determination. Theevaluation for the subject may be determined in response to the data ofthe subject. The categorical determination may be selected from thegroup consisting of a presence of the cognitive function attribute andan absence of the cognitive function attribute. The cognitive functionattribute may be determined with a sensitivity of at least 80 and aspecificity of at least 80, respectively, for the presence or theabsence of the cognitive function attribute. The cognitive functionattribute may be determined with a sensitivity of at least 90 and aspecificity of at least 90, respectively, for the presence or theabsence of the cognitive function attribute. The cognitive functionattribute may comprise a behavioral disorder and a developmentaladvancement.

In another aspect, a digital therapeutic system to treat a subject witha personal therapeutic treatment plan may comprise one or moreprocessors, a diagnostic module to receive data from the subject andoutput diagnostic data for the subject, and a therapeutic module toreceive the diagnostic data and output the personal therapeutictreatment plan for the subject. The diagnostic module may comprise oneor more classifiers built using machine learning or statistical modelingbased on a subject population to determine the diagnostic data for thesubject. The diagnostic data may comprise an evaluation for the subject,the evaluation selected from the group consisting of an inconclusivedetermination and a categorical determination in response to datareceived from the subject. The therapeutic module may comprise one ormore models built using machine learning or statistical modeling basedon at least a portion the subject population to determine and output thepersonal therapeutic treatment plan of the subject. The diagnosticmodule may be configured to receive updated subject data from thesubject in response to therapy of the subject and generate updateddiagnostic data from the subject. The therapeutic module may beconfigured to receive the updated diagnostic data and output an updatedpersonal treatment plan for the subject in response to the diagnosticdata and the updated diagnostic data. The diagnostic module may comprisea diagnostic machine learning classifier trained on the subjectpopulation. The therapeutic module may comprise a therapeutic machinelearning classifier trained on the at least the portion of the subjectpopulation. The diagnostic module and the therapeutic module may bearranged for the diagnostic module to provide feedback to thetherapeutic module based on performance of the treatment plan. Thetherapeutic classifier may comprise instructions trained on a data setcomprising a population of which the subject is not a member. Thesubject may comprise a person who is not a member of the population. Thediagnostic module may comprise a diagnostic classifier trained onplurality of profiles of a subject population of at least 10,000 peopleand therapeutic profile trained on the plurality of profiles of thesubject population.

In another aspect, a digital therapeutic system to treat a subject witha personal therapeutic treatment plan may comprise a processor, adiagnostic module to receive data from the subject and output diagnosticdata for the subject, and a therapeutic module to receive the diagnosticdata and output the personal therapeutic treatment plan for the subject.

The diagnostic data may comprise an evaluation for the subject, theevaluation selected from the group consisting of an inconclusivedetermination and a categorical determination in response to datareceived from the subject. The personal therapeutic treatment plan maycomprise digital therapeutics. The digital therapeutics may compriseinstructions, feedback, activities or interactions provided to thesubject or caregiver. The digital therapeutics may be provided with amobile device. The diagnostics data and the personal therapeutictreatment plan may be provided to a third-party system. The third-partysystem may comprise a computer system of a health care professional or atherapeutic delivery system. The diagnostic module may be configured toreceive updated subject data from the subject in response to a feedbackdata of the subject and generate updated diagnostic data. Thetherapeutic module may be configured to receive the updated diagnosticdata and output an updated personal treatment plan for the subject inresponse to the diagnostic data and the updated diagnostic data. Theupdated subject data may be received in response to a feedback data thatidentifies relative levels of efficacy, compliance and responseresulting from the personal therapeutic treatment plan. The diagnosticmodule may use machine learning or statistical modeling based on asubject population to determine the diagnostic data. The therapeuticmodule may be based on at least a portion the subject population todetermine the personal therapeutic treatment plan of the subject. Thediagnostic module may comprise a diagnostic machine learning classifiertrained on a subject population. The therapeutic module may comprise atherapeutic machine learning classifier trained on at least a portion ofthe subject population. The diagnostic module may be configured toprovide feedback to the therapeutic module based on performance of thepersonal therapeutic treatment plan. The data from the subject maycomprise at least one of the subject and caregiver video, audio,responses to questions or activities, and active or passive data streamsfrom user interaction with activities, games or software features of thesystem. The subject may have a risk selected from the group consistingof a behavioral disorder, neurological disorder and mental healthdisorder. The behavioral, neurological or mental health disorder may beselected from the group consisting of autism, autistic spectrum,attention deficit disorder, depression, obsessive compulsive disorder,schizophrenia, Alzheimer's disease, dementia, attention deficithyperactive disorder and speech and learning disability. The diagnosticmodule may be configured for an adult to perform an assessment orprovide data for an assessment of a child or juvenile. The diagnosticmodule may be configured for a caregiver or family member to perform anassessment or provide data for an assessment of the subject.

In another aspect, a non-transitory computer-readable storage media maybe encoded with a program. The computer program may include executableinstructions for: 1) receiving input data from the subject andoutputting diagnostic data for the subject; 2) receiving the diagnosticdata and outputting a personal therapeutic treatment plan for thesubject; and 3) evaluating the diagnostic data based on at least aportion the subject population to determine and output the personaltherapeutic treatment plan of the subject. The diagnostic data maycomprise an evaluation for the subject, the evaluation selected from thegroup consisting of an inconclusive determination and a categoricaldetermination in response to input data received from the subject.Updated subject input data may be received from the subject in responseto therapy of the subject and updated diagnostic data may be generatedfrom the subject. Updated diagnostic data may be received and an updatedpersonal treatment plan may be outputted for the subject in response tothe diagnostic data and the updated diagnostic data.

In another aspect, a non-transitory computer-readable storage media maybe encoded with a computer program. The computer program may includeexecutable instructions for receiving input data from a subject andoutputting diagnostic data for the subject and receiving the diagnosticdata and outputting a personal therapeutic treatment plan for thesubject. The diagnostic data may comprise an evaluation for the subject,the evaluation selected from the group consisting of an inconclusivedetermination and a categorical determination in response to datareceived from the subject. The personal therapeutic treatment plan maycomprise digital therapeutics.

In another aspect, a method of treating a subject with a personaltherapeutic treatment plan may comprise a diagnostic process ofreceiving data from the subject and outputting diagnostic data for thesubject wherein the diagnostic data comprises an evaluation for thesubject and a therapeutic process of receiving the diagnostic data andoutputting the personal therapeutic treatment plan for the subject. Theevaluation may be selected from the group consisting of an inconclusivedetermination and a categorical determination in response to datareceived from the subject. The diagnostic process may comprise receivingupdated subject data from the subject in response to a therapy of thesubject and generating an updated diagnostic data from the subject. Thetherapeutic process may comprise receiving the updated diagnostic dataand outputting an updated personal treatment plan for the subject inresponse to the diagnostic data and the updated diagnostic data. Theupdated subject data may be received in response to a feedback data thatidentifies relative levels of efficacy, compliance and responseresulting from the personal therapeutic treatment plan. The personaltherapeutic treatment plan may comprise digital therapeutics. Thedigital therapeutics may comprise instructions, feedback, activities orinteractions provided to the subject or caregiver. The digitaltherapeutics may be provided with a mobile device. The method mayfurther comprise providing the diagnostics data and the personaltherapeutic treatment plan to a third-party system. The third-partysystem may comprise a computer system of a health care professional or atherapeutic delivery system. The diagnostic process may be performed bya process selected from the group consisting of machine learning, aclassifier, artificial intelligence, or statistical modeling based on asubject population to determine the diagnostic data. The therapeuticprocess may be performed by a process selected from the group consistingof machine learning, a classifier, artificial intelligence, orstatistical modeling based on at least a portion the subject populationto determine the personal therapeutic treatment plan of the subject. Thediagnostic process may be performed by a diagnostic machine learningclassifier trained on a subject population. The therapeutic process maybe performed by a therapeutic machine learning classifier trained on atleast a portion of the subject population. The diagnostic process maycomprise providing feedback to the therapeutic module based onperformance of the personal therapeutic treatment plan. The data fromthe subject may comprise at least one of the subject and caregivervideo, audio, responses to questions or activities, and active orpassive data streams from user interaction with activities, games orsoftware features. The diagnostic process may be performed by an adultto perform an assessment or provide data for an assessment of a child orjuvenile. The diagnostic process may enable a caregiver or family memberto perform an assessment or provide data for an assessment of thesubject. The subject may have a risk selected from the group consistingof a behavioral disorder, neurological disorder, and mental healthdisorder. The risk may be selected from the group consisting of autism,autistic spectrum, attention deficit disorder, depression, obsessivecompulsive disorder, schizophrenia, Alzheimer's disease, dementia,attention deficit hyperactive disorder, and speech and learningdisability.

FIGS. 1A and 1B show some exemplary developmental disorders that may beevaluated using the assessment procedure as described herein. Theassessment procedure can be configured to evaluate a subject's risk forhaving one or more developmental disorders, such as two or more relateddevelopmental disorders. The developmental disorders may have at leastsome overlap in symptoms or features of the subject. Such developmentaldisorders may include pervasive development disorder (PDD), autismspectrum disorder (ASD), social communication disorder, restrictedrepetitive behaviors, interests, and activities (RRBs), autism(“classical autism”), Asperger's Syndrome (“high functioning autism),PDD-not otherwise specified (PDD-NOS, “atypical autism”), attentiondeficit and hyperactivity disorder (ADHD), speech and language delay,obsessive compulsive disorder (OCD), intellectual disability, learningdisability, or any other relevant development disorder, such asdisorders defined in any edition of the Diagnostic and StatisticalManual of Mental Disorders (DSM). The assessment procedure may beconfigured to determine the risk of the subject for having each of aplurality of disorders. The assessment procedure may be configured todetermine the subject as at greater risk of a first disorder or a seconddisorder of the plurality of disorders. The assessment procedure may beconfigured to determine the subject as at risk of a first disorder and asecond disorder with comorbidity. The assessment procedure may beconfigured to predict a subject to have normal development, or have lowrisk of having any of the disorders the procedure is configured toscreen for. The assessment procedure may further be configured to havehigh sensitivity and specificity to distinguish among different severityratings for a disorder; for example, the procedure may be configured topredict a subject's risk for having level 1 ASD, level 2 ASD, or level 3ASD as defined in the fifth edition of the DSM (DSM-V).

Many developmental disorders may have similar or overlapping symptoms,thus complicating the assessment of a subject's developmental disorder.The assessment procedure described herein can be configured to evaluatea plurality of features of the subject that may be relevant to one ormore developmental disorders. The procedure can comprise an assessmentmodel that has been trained using a large set of clinically validateddata to learn the statistical relationship between a feature of asubject and clinical diagnosis of one or more developmental disorders.Thus, as a subject participates in the assessment procedure, thesubject's feature value for each evaluated feature (e.g., subject'sanswer to a question) can be queried against the assessment model toidentify the statistical correlation, if any, of the subject's featurevalue to one or more screened developmental disorders. Based on thefeature values provided by the subject, and the relationship betweenthose values and the predicted risk for one or more developmentaldisorders as determined by the assessment model, the assessmentprocedure can dynamically adjust the selection of next features to beevaluated in the subject. The selection of the next feature to beevaluated may comprise an identification of the next most predictivefeature, based on the determination of the subject as at risk for aparticular disorder of the plurality of disorders being screened. Forexample, if after the subject has answered the first five questions ofthe assessment procedure, the assessment model predicts a low risk ofautism and a relatively higher risk of ADHD in the subject, theassessment procedure may select features with higher relevance to ADHDto be evaluated next in the subject (e.g., questions whose answers arehighly correlated with a clinical diagnosis of ADHD may be presentednext to the subject). Thus, the assessment procedure described hereincan be dynamically tailored to a particular subject's risk profile, andenable the evaluation of the subject's disorder with a high level ofgranularity.

FIG. 2 is a schematic diagram of an exemplary data processing module 100for providing the assessment procedure as described herein. The dataprocessing module 100 generally comprises a preprocessing module 105, atraining module 110, and a prediction module 120. The data processingmodule can extract training data 150 from a database, or intake new data155 with a user interface 130. The preprocessing module can apply one ormore transformations to standardize the training data or new data forthe training module or the prediction module. The preprocessed trainingdata can be passed to the training module, which can construct anassessment model 160 based on the training data. The training module mayfurther comprise a validation module 115, configured to validate thetrained assessment model using any appropriate validation algorithm(e.g., Stratified K-fold cross-validation). The preprocessed new datacan be passed on to the prediction module, which may output a prediction170 of the subject's developmental disorder by fitting the new data tothe assessment model constructed in the training module. The predictionmodule may further comprise a feature recommendation module 125,configured to select or recommend the next feature to be evaluated inthe subject, based on previously provided feature values for thesubject.

The training data 150, used by the training module to construct theassessment model, can comprise a plurality of datasets from a pluralityof subjects, each subject's dataset comprising an array of features andcorresponding feature values, and a classification of the subject'sdevelopmental disorder or condition. As described herein, the featuresmay be evaluated in the subject via one or more of questions asked tothe subject, observations of the subject, or structured interactionswith the subject. Feature values may comprise one or more of answers tothe questions, observations of the subject such as characterizationsbased on video images, or responses of the subject to a structuredinteraction, for example. Each feature may be relevant to theidentification of one or more developmental disorders or conditions, andeach corresponding feature value may indicate the degree of presence ofthe feature in the specific subject. For example, a feature may be theability of the subject to engage in imaginative or pretend play, and thefeature value for a particular subject may be a score of either 0, 1, 2,3, or 8, wherein each score corresponds to the degree of presence of thefeature in the subject (e.g., 0=variety of pretend play; 1=some pretendplay; 2=occasional pretending or highly repetitive pretend play; 3=nopretend play; 8=not applicable). The feature may be evaluated in thesubject by way of a question presented to the subject or a caretakersuch as a parent, wherein the answer to the question comprises thefeature value. Alternatively or in combination, the feature may beobserved in the subject, for example with a video of the subjectengaging in a certain behavior, and the feature value may be identifiedthrough the observation. In addition to the array of features andcorresponding feature values, each subject's dataset in the trainingdata also comprises a classification of the subject. For example, theclassification may be autism, autism spectrum disorder (ASD), ornon-spectrum. Preferably, the classification comprises a clinicaldiagnosis, assigned by qualified personnel such as licensed clinicalpsychologists, in order to improve the predictive accuracy of thegenerated assessment model. The training data may comprise datasetsavailable from large data repositories, such as Autism DiagnosticInterview-Revised (ADI-R) data and/or Autism Diagnostic ObservationSchedule (ADOS) data available from the Autism Genetic Resource Exchange(AGRE), or any datasets available from any other suitable repository ofdata (e.g., Boston Autism Consortium (AC), Simons Foundation, NationalDatabase for Autism Research, etc.). Alternatively or in combination,the training data may comprise large self-reported datasets, which canbe crowd-sourced from users (e.g., via websites, mobile applications,etc.).

The preprocessing module 105 can be configured to apply one or moretransformations to the extracted training data to clean and normalizethe data, for example. The preprocessing module can be configured todiscard features which contain spurious metadata or contain very fewobservations. The preprocessing module can be further configured tostandardize the encoding of feature values. Different datasets may oftenhave the same feature value encoded in different ways, depending on thesource of the dataset. For example, ‘900’, ‘900.0’, ‘904’, ‘904.0’,‘−1’, ‘−1.0’, ‘None’, and ‘NaN’ may all encode for a “missing” featurevalue. The preprocessing module can be configured to recognize theencoding variants for the same feature value, and standardize thedatasets to have a uniform encoding for a given feature value. Thepreprocessing module can thus reduce irregularities in the input datafor the training and prediction modules, thereby improving therobustness of the training and prediction modules.

In addition to standardizing data, the preprocessing module can also beconfigured to re-encode certain feature values into a different datarepresentation. In some instances, the original data representation ofthe feature values in a dataset may not be ideal for the construction ofan assessment model. For example, for a categorical feature wherein thecorresponding feature values are encoded as integers from 1 to 9, eachinteger value may have a different semantic content that is independentof the other values. For example, a value of ‘1’ and a value of ‘9’ mayboth be highly correlated with a specific classification, while a valueof ‘5’ is not. The original data representation of the feature value,wherein the feature value is encoded as the integer itself, may not beable to capture the unique semantic content of each value, since thevalues are represented in a linear model (e.g., an answer of ‘5’ wouldplace the subject squarely between a ‘1’ and a ‘9’ when the feature isconsidered in isolation; however, such an interpretation would beincorrect in the aforementioned case wherein a ‘1’ and a ‘9’ are highlycorrelated with a given classification while a ‘5’ is not). To ensurethat the semantic content of each feature value is captured in theconstruction of the assessment model, the preprocessing module maycomprise instructions to re-encode certain feature values, such asfeature values corresponding to categorical features, in a “one-hot”fashion, for example. In a “one-hot” representation, a feature value maybe represented as an array of bits having a value of 0 or 1, the numberof bits corresponding to the number of possible values for the feature.Only the feature value for the subject may be represented as a “1”, withall other values represented as a “0”. For example, if a subjectanswered “4” to a question whose possible answers comprise integers from1 to 9, the original data representation may be [4], and the one-hotrepresentation may be [0 0 0 1 0 0 0 0 0]. Such a one-hot representationof feature values can allow every value to be considered independentlyof the other possible values, in cases where such a representation wouldbe necessary. By thus re-encoding the training data using the mostappropriate data representation for each feature, the preprocessingmodule can improve the accuracy of the assessment model constructedusing the training data.

The preprocessing module can be further configured to impute any missingdata values, such that downstream modules can correctly process thedata. For example, if a training dataset provided to the training modulecomprises data missing an answer to one of the questions, thepreprocessing module can provide the missing value, so that the datasetcan be processed correctly by the training module. Similarly, if a newdataset provided to the prediction module is missing one or more featurevalues (e.g., the dataset being queried comprises only the answer to thefirst question in a series of questions to be asked), the preprocessingmodule can provide the missing values, so as to enable correctprocessing of the dataset by the prediction module. For features havingcategorical feature values (e.g., extent of display of a certainbehavior in the subject), missing values can be provided as appropriatedata representations specifically designated as such. For example, ifthe categorical features are encoded in a one-hot representation asdescribed herein, the preprocessing module may encode a missingcategorical feature value as an array of ‘0’ bits. For features havingcontinuous feature values (e.g., age of the subject), the mean of all ofthe possible values can be provided in place of the missing value (e.g.,age of 4 years).

The training module 110 can utilize a machine learning algorithm orother algorithm to construct and train an assessment model to be used inthe assessment procedure, for example. An assessment model can beconstructed to capture, based on the training data, the statisticalrelationship, if any, between a given feature value and a specificdevelopmental disorder to be screened by the assessment procedure. Theassessment model may, for example, comprise the statistical correlationsbetween a plurality of clinical characteristics and clinical diagnosesof one or more developmental disorders. A given feature value may have adifferent predictive utility for classifying each of the plurality ofdevelopmental disorders to be evaluated in the assessment procedure. Forexample, in the aforementioned example of a feature comprising theability of the subject to engage in imaginative or pretend play, thefeature value of “3” or “no variety of pretend play” may have a highpredictive utility for classifying autism, while the same feature valuemay have low predictive utility for classifying ADHD. Accordingly, foreach feature value, a probability distribution may be extracted thatdescribes the probability of the specific feature value for predictingeach of the plurality of developmental disorders to be screened by theassessment procedure. The machine learning algorithm can be used toextract these statistical relationships from the training data and buildan assessment model that can yield an accurate prediction of adevelopmental disorder when a dataset comprising one or more featurevalues is fitted to the model.

One or more machine learning algorithms may be used to construct theassessment model, such as support vector machines that deploy stepwisebackwards feature selection and/or graphical models, both of which canhave advantages of inferring interactions between features. For example,machine learning algorithms or other statistical algorithms may be used,such as alternating decision trees (ADTree), Decision Stumps, functionaltrees (FT), logistic model trees (LMT), logistic regression, RandomForests, linear classifiers, or any machine learning algorithm orstatistical algorithm known in the art. One or more algorithms may beused together to generate an ensemble method, wherein the ensemblemethod may be optimized using a machine learning ensemble meta-algorithmsuch as a boosting (e.g., AdaBoost, LPBoost, TotalBoost, BrownBoost,MadaBoost, LogitBoost, etc.) to reduce bias and/or variance. Once anassessment model is derived from the training data, the model may beused as a prediction tool to assess the risk of a subject for having oneor more developmental disorders. Machine learning analyses may beperformed using one or more of many programming languages and platformsknown in the art, such as R, Weka, Python, and/or Matlab, for example.

A Random Forest classifier, which generally comprises a plurality ofdecision trees wherein the output prediction is the mode of thepredicted classifications of the individual trees, can be helpful inreducing overfitting to training data. An ensemble of decision trees canbe constructed using a random subset of features at each split ordecision node. The Gini criterion may be employed to choose the bestpartition, wherein decision nodes having the lowest calculated Giniimpurity index are selected. At prediction time, a “vote” can be takenover all of the decision trees, and the majority vote (or mode of thepredicted classifications) can be output as the predictedclassification.

FIG. 3 is a schematic diagram illustrating a portion of an exemplaryassessment model 160 based on a Random Forest classifier. The assessmentmodule may comprise a plurality of individual decision trees 165, suchas decision trees 165 a and 165 b, each of which can be generatedindependently using a random subset of features in the training data.Each decision tree may comprise one or more decision nodes such asdecision nodes 166 and 167 shown in FIG. 3, wherein each decision nodespecifies a predicate condition. For example, decision node 16predicates the condition that, for a given dataset of an individual, theanswer to ADI-R question #86 (age when abnormality is first evident) is4 or less. Decision node 167 predicates the condition that, for thegiven dataset, the answer to ADI-R question #52 (showing and directionattention) is 8 or less. At each decision node, a decision tree can besplit based on whether the predicate condition attached to the decisionnode holds true, leading to prediction nodes (e.g., 166 a, 166 b, 167 a,167 b). Each prediction node can comprise output values (‘value’ in FIG.3) that represent “votes” for one or more of the classifications orconditions being evaluated by the assessment model. For example, in theprediction nodes shown in FIG. 3, the output values comprise votes forthe individual being classified as having autism or being non-spectrum.A prediction node can lead to one or more additional decision nodesdownstream (not shown in FIG. 3), each decision node leading to anadditional split in the decision tree associated with correspondingprediction nodes having corresponding output values. The Gini impuritycan be used as a criterion to find informative features based on whichthe splits in each decision tree may be constructed.

When the dataset being queried in the assessment model reaches a “leaf”,or a final prediction node with no further downstream splits, the outputvalues of the leaf can be output as the votes for the particulardecision tree. Since the Random Forest model comprises a plurality ofdecision trees, the final votes across all trees in the forest can besummed to yield the final votes and the corresponding classification ofthe subject. While only two decision trees are shown in FIG. 3, themodel can comprise any number of decision trees. A large number ofdecision trees can help reduce overfitting of the assessment model tothe training data, by reducing the variance of each individual decisiontree. For example, the assessment model can comprise at least about 10decision trees, for example at least about 100 individual decision treesor more.

An ensemble of linear classifiers may also be suitable for thederivation of an assessment model as described herein. Each linearclassifier can be individually trained with a stochastic gradientdescent, without an “intercept term”. The lack of an intercept term canprevent the classifier from deriving any significance from missingfeature values. For example, if a subject did not answer a question suchthat the feature value corresponding to said question is represented asan array of ‘0’ bits in the subject's data set, the linear classifiertrained without an intercept term will not attribute any significance tothe array of ‘0’ bits. The resultant assessment model can thereby avoidestablishing a correlation between the selection of features orquestions that have been answered by the subject and the finalclassification of the subject as determined by the model. Such analgorithm can help ensure that only the subject-provided feature valuesor answers, rather than the features or questions, are factored into thefinal classification of the subject.

The training module may comprise feature selection. One or more featureselection algorithms (such as support vector machine, convolutionalneural nets) may be used to select features able to differentiatebetween individuals with and without certain developmental disorders.Different sets of features may be selected as relevant for theidentification of different disorders. Stepwise backwards algorithms maybe used along with other algorithms. The feature selection procedure mayinclude a determination of an optimal number of features.

The training module may be configured to evaluate the performance of thederived assessment models. For example, the accuracy, sensitivity, andspecificity of the model in classifying data can be evaluated. Theevaluation can be used as a guideline in selecting suitable machinelearning algorithms or parameters thereof. The training module can thusupdate and/or refine the derived assessment model to maximize thespecificity (the true negative rate) over sensitivity (the true positiverate). Such optimization may be particularly helpful when classimbalance or sample bias exists in training data.

In at least some instances, available training data may be skewedtowards individuals diagnosed with a specific developmental disorder. Insuch instances, the training data may produce an assessment modelreflecting that sample bias, such that the model assumes that subjectsare at risk for the specific developmental disorder unless there is astrong case to be made otherwise. An assessment model incorporating sucha particular sample bias can have less than ideal performance ingenerating predictions of new or unclassified data, since the new datamay be drawn from a subject population which may not comprise a samplebias similar to that present in the training data. To reduce sample biasin constructing an assessment model using skewed training data, sampleweighting may be applied in training the assessment model. Sampleweighting can comprise lending a relatively greater degree ofsignificance to a specific set of samples during the model trainingprocess. For example, during model training, if the training data isskewed towards individuals diagnosed with autism, higher significancecan be attributed to the data from individuals not diagnosed with autism(e.g., up to 50 times more significance than data from individualsdiagnosed with autism). Such a sample weighting technique cansubstantially balance the sample bias present in the training data,thereby producing an assessment model with reduced bias and improvedaccuracy in classifying data in the real world. To further reduce thecontribution of training data sample bias to the generation of anassessment model, a boosting technique may be implemented during thetraining process. Boosting comprises an iterative process, wherein afterone iteration of training, the weighting of each sample data point isupdated. For example, samples that are misclassified after the iterationcan be updated with higher significances. The training process may thenbe repeated with the updated weightings for the training data.

The training module may further comprise a validation module 115configured to validate the assessment model constructed using thetraining data. For example, a validation module may be configured toimplement a Stratified K-fold cross validation, wherein k represents thenumber of partitions that the training data is split into for crossvalidation. For example, k can be any integer greater than 1, such as 3,4, 5, 6, 7, 8, 9, or 10, or possibly higher depending on risk ofoverfitting the assessment model to the training data.

The training module may be configured to save a trained assessment modelto a local memory and/or a remote server, such that the model can beretrieved for modification by the training module or for the generationof a prediction by the prediction module 120.

FIG. 4 is an exemplary operational flow 400 of a method of a predictionmodule 120 as described herein. The prediction module 120 can beconfigured to generate a predicted classification (e.g., developmentaldisorder) of a given subject, by fitting new data to an assessment modelconstructed in the training module. At step 405, the prediction modulecan receive new data that may have been processed by the preprocessingmodule to standardize the data, for example by dropping spuriousmetadata, applying uniform encoding of feature values, re-encodingselect features using different data representations, and/or imputingmissing data points, as described herein. The new data can comprise anarray of features and corresponding feature values for a particularsubject. As described herein, the features may comprise a plurality ofquestions presented to a subject, observations of the subject, or tasksassigned to the subject. The feature values may comprise input data fromthe subject corresponding to characteristics of the subject, such asanswers of the subject to questions asked, or responses of the subject.The new data provided to the prediction module may or may not have aknown classification or diagnosis associated with the data; either way,the prediction module may not use any pre-assigned classificationinformation in generating the predicted classification for the subject.The new data may comprise a previously-collected, complete dataset for asubject to be diagnosed or assessed for the risk of having one or moreof a plurality of developmental disorders. Alternatively or incombination, the new data may comprise data collected in real time fromthe subject or a caretaker of the subject, for example with a userinterface as described in further detail herein, such that the completedataset can be populated in real time as each new feature value providedby the subject is sequentially queried against the assessment model.

At step 410, the prediction module can load a previously savedassessment model, constructed by the training module, from a localmemory and/or a remote server configured to store the model. At step415, the new data is fitted to the assessment model to generate apredicted classification of the subject. At step 420, the module cancheck whether the fitting of the data can generate a prediction of oneor more specific disorders (e.g., autism, ADHD, etc.) within aconfidence interval exceeding a threshold value, for example within a90% or higher confidence interval, for example 95% or more. If so, asshown in step 425, the prediction module can output the one or moredevelopmental disorders as diagnoses of the subject or as disorders forwhich the subject is at risk. The prediction module may output aplurality of developmental disorders for which the subject is determinedto at risk beyond the set threshold, optionally presenting the pluralityof disorders in order of risk. The prediction module may output onedevelopmental disorder for which the subject is determined to be atgreatest risk. The prediction module may output two or more developmentdisorders for which the subject is determined to risk with comorbidity.The prediction module may output determined risk for each of the one ormore developmental disorders in the assessment model. If the predictionmodule cannot fit the data to any specific developmental disorder withina confidence interval at or exceeding the designated threshold value,the prediction module may determine, in step 430, whether there are anyadditional features that can be queried. If the new data comprises apreviously-collected, complete dataset, and the subject cannot bequeried for any additional feature values, “no diagnosis” may be outputas the predicted classification, as shown in step 440. If the new datacomprises data collected in real time from the subject or caretakerduring the prediction process, such that the dataset is updated witheach new input data value provided to the prediction module and eachupdated dataset is fitted to the assessment model, the prediction modulemay be able to query the subject for additional feature values. If theprediction module has already obtained data for all features included inthe assessment module, the prediction module may output “no diagnosis”as the predicted classification of the subject, as shown in step 440. Ifthere are features that have not yet been presented to the subject, asshown in step 435, the prediction module may obtain additional inputdata values from the subject, for example by presenting additionalquestions to the subject. The updated dataset including the additionalinput data may then be fitted to the assessment model again (step 415),and the loop may continue until the prediction module can generate anoutput.

FIG. 5 is an exemplary operational flow 500 of a feature recommendationmodule 125 as described herein by way of a non-limiting example. Theprediction module may comprise a feature recommendation module 125,configured to identify, select or recommend the next most predictive orrelevant feature to be evaluated in the subject, based on previouslyprovided feature values for the subject. For example, the featurerecommendation module can be a question recommendation module, whereinthe module can select the most predictive next question to be presentedto a subject or caretaker, based on the answers to previously presentedquestions. The feature recommendation module can be configured torecommend one or more next questions or features having the highestpredictive utility in classifying a particular subject's developmentaldisorder. The feature recommendation module can thus help to dynamicallytailor the assessment procedure to the subject, so as to enable theprediction module to produce a prediction with a reduced length ofassessment and improved sensitivity and accuracy. Further, the featurerecommendation module can help improve the specificity of the finalprediction generated by the prediction module, by selecting features tobe presented to the subject that are most relevant in predicting one ormore specific developmental disorders that the particular subject ismost likely to have, based on feature values previously provided by thesubject.

At step 505, the feature recommendation module can receive as input thedata already obtained from the subject in the assessment procedure. Theinput subject data can comprise an array of features and correspondingfeature values provided by the subject. At step 510, the featurerecommendation module can select one or more features to be consideredas “candidate features” for recommendation as the next feature(s) to bepresented to one or more of the subject, caretaker or clinician.Features that have already been presented can be excluded from the groupof candidate features to be considered. Optionally, additional featuresmeeting certain criteria may also be excluded from the group ofcandidate features, as described in further detail herein.

At step 515, the feature recommendation module can evaluate the“expected feature importance” of each candidate feature. The candidatefeatures can be evaluated for their “expected feature importance”, orthe estimated utility of each candidate feature in predicting a specificdevelopmental disorder for the specific subject. The featurerecommendation module may utilize an algorithm based on: (1) theimportance or relevance of a specific feature value in predicting aspecific developmental disorder; and (2) the probability that thesubject may provide the specific feature value. For example, if theanswer of “3” to ADOS question B5 is highly correlated with aclassification of autism, this answer can be considered a feature valuehaving high utility for predicting autism. If the subject at hand alsohas a high probability of answering “3” to said question B5, the featurerecommendation module can determine this question to have high expectedfeature importance. An algorithm that can be used to determine theexpected feature importance of a feature is described in further detailin reference to FIG. 6, for example.

At step 520, the feature recommendation module can select one or morecandidate features to be presented next to the subject, based on theexpected feature importance of the features as determined in step 515.For example, the expected feature importance of each candidate featuremay be represented as a score or a real number, which can then be rankedin comparison to other candidate features. The candidate feature havingthe desired rank, for example a top 10, top 5, top 3, top 2, or thehighest rank, may be selected as the feature to the presented next tothe subject.

FIG. 6 is an exemplary operational flow 600 of method of determining anexpected feature importance determination algorithm 127 as performed bya feature recommendation module 125 described herein.

At step 605, the algorithm can determine the importance or relevance ofa specific feature value in predicting a specific developmentaldisorder. The importance or relevance of a specific feature value inpredicting a specific developmental disorder can be derived from theassessment model constructed using training data. Such a “feature valueimportance” can be conceptualized as a measure of how relevant a givenfeature value's role is, should it be present or not present, indetermining a subject's final classification. For example, if theassessment model comprises a Random Forest classifier, the importance ofa specific feature value can be a function of where that feature ispositioned in the Random Forest classifier's branches. Generally, if theaverage position of the feature in the decision trees is relativelyhigh, the feature can have relatively high feature importance. Theimportance of a feature value given a specific assessment model can becomputed efficiently, either by the feature recommendation module or bythe training module, wherein the training module may pass the computedstatistics to the feature recommendation module. Alternatively, theimportance of a specific feature value can be a function of the actualprediction confidence that would result if said feature value wasprovided by the subject. For each possible feature value for a givencandidate feature, the feature recommendation module can be configuredto calculate the actual prediction confidence for predicting one or moredevelopmental disorders, based on the subject's previously providedfeature values and the currently assumed feature value.

Each feature value may have a different importance for eachdevelopmental disorder for which the assessment procedure is designed toscreen. Accordingly, the importance of each feature value may berepresented as a probability distribution that describes the probabilityof the feature value yielding an accurate prediction for each of theplurality of developmental disorders being evaluated.

At step 610, the feature recommendation module can determine theprobability of a subject providing each feature value. The probabilitythat the subject may provide a specific feature value can be computedusing any appropriate statistical model. For example, a largeprobabilistic graphical model can be used to find the values ofexpressions such as:

prob(E=1|A=1,B=2,C=1)

where A, B, and C represent different features or questions in theprediction module and the integers 1 and 2 represent different possiblefeature values for the feature (or possible answers to the questions).The probability of a subject providing a specific feature value may thenbe computed using Bayes' rule, with expressions such as:

prob(E=1|A=1,B=2,C=1)=prob(E=1,A=1,B=2,C=1)/prob(A=1,B=2,C=1)

Such expressions may be computationally expensive, in terms of bothcomputation time and required processing resources. Alternatively or incombination with computing the probabilities explicitly using Bayes'rule, logistic regression or other statistical estimators may be used,wherein the probability is estimated using parameters derived from amachine learning algorithm. For example, the following expression may beused to estimate the probability that the subject may provide a specificfeature value:

prob(E=1|A=1,B=2,C=1)sigmoid(a1*A+a2*B+a3*C+a4),

wherein a1, a2, a3, and a4 are constant coefficients determined from thetrained assessment model, learned using an optimization algorithm thatattempts to make this expression maximally correct, and wherein sigmoidis a nonlinear function that enables this expression to be turned into aprobability. Such an algorithm can be quick to train, and the resultingexpressions can be computed quickly in application, e.g., duringadministration of the assessment procedure. Although reference is madeto four coefficients, as many coefficients as are helpful may be used aswill be recognized by a person of ordinary skill in the art.

At step 615, the expected importance of each feature value can bedetermined based on a combination of the metrics calculated in steps 605and 610. Based on these two factors, the feature recommendation modulecan determine the expected utility of the specific feature value inpredicting a specific developmental disorder. Although reference is madeherein to the determination of expected importance via multiplication,the expected importance can be determined by combining coefficients andparameters in many ways, such as with look up tables, logic, ordivision, for example.

At step 620, steps 605-615 can be repeated for every possible featurevalue for each candidate feature. For example, if a particular questionhas 4 possible answers, the expected importance of each of the 4possible answers is determined.

At step 625, the total expected importance, or the expected featureimportance, of each candidate feature can be determined. The expectedfeature importance of each feature can be determined by summing thefeature value importances of every possible feature value for thefeature, as determined in step 620. By thus summing the expectedutilities across all possible feature values for a given feature, thefeature recommendation module can determine the total expected featureimportance of the feature for predicting a specific developmentaldisorder in response to previous answers.

At step 630, steps 605-625 can be repeated for every candidate featurebeing considered by the feature recommendation module. The candidatefeatures may comprise a subset of possible features such as questions.Thus, an expected feature importance score for every candidate featurecan be generated, and the candidate features can be ranked in order ofhighest to lowest expected feature importance.

Optionally, in addition to the two factors determined in steps 605 and610, a third factor may also be taken into account in determining theimportance of each feature value. Based on the subject's previouslyprovided feature values, the subject's probability of having one or moreof the plurality of developmental disorders can be determined. Such aprobability can be determined based on the probability distributionstored in the assessment model, indicating the probability of thesubject having each of the plurality of screened developmental disordersbased on the feature values provided by the subject. In selecting thenext feature to be presented to the subject, the algorithm may beconfigured to give greater weight to the feature values most importantor relevant to predicting the one or more developmental disorders thatthe subject at hand is most likely to have. For example, if a subject'spreviously provided feature values indicate that the subject has ahigher probability of having either an intellectual disability or speechand language delay than any of the other developmental disorders beingevaluated, the feature recommendation module can favor feature valueshaving high importance for predicting either intellectual disability orspeech and language delay, rather than features having high importancefor predicting autism, ADHD, or any other developmental disorder thatthe assessment is designed to screen for. The feature recommendationmodule can thus enable the prediction module to tailor the predictionprocess to the subject at hand, presenting more features that arerelevant to the subject's potential developmental disorder to yield afinal classification with higher granularity and confidence.

Although the above steps show an exemplary operational flow 600 of anexpected feature importance determination algorithm 127, a person ofordinary skill in the art will recognize many variations based on theteachings described herein. The steps may be completed in a differentorder. Steps may be added or deleted. Some of the steps may comprisesub-steps of other steps. Many of the steps may be repeated as often asdesired by the user.

An exemplary implementation of the feature recommendation module is nowdescribed. Subject X has provided answers (feature values) to questions(features) A, B, and C in the assessment procedure:

Subject X={‘A’:1,‘B’:2,‘C’:1}

The feature recommendation module can determine whether question D orquestion E should be presented next in order to maximally increase thepredictive confidence with which a final classification or diagnosis canbe reached. Given Subject X's previous answers, the featurerecommendation module determines the probability of Subject X providingeach possible answer to each of questions D and E, as follows:

prob(E=1|A=1,B=2,C=1)=0.1

prob(E=2|A=1,B=2,C=1)=0.9

prob(D=1|A=1,B=2,C=1)=0.7

prob(D=2|A=1,B=2,C=1)=0.3

The feature importance of each possible answer to each of questions Dand E can be computed based on the assessment model as described.Alternatively, the feature importance of each possible answer to each ofquestions D and E can be computed as the actual prediction confidencethat would result if the subject were to give the specific answer. Theimportance of each answer can be represented using a range of values onany appropriate numerical scale. For example:

importance(E=1)=1

importance(E=2)=3

importance(D=1)=2

importance(D=2)=4

Based on the computed probabilities and the feature value importances,the feature recommendation module can compute the expected featureimportance of each question as follows:

Expectation[importance(E)]=(prob(E=1|A=1,B=2,C=1)*importance(E=1)+(prob(E=2|A=1,B=2,C=1)*importance(E=2)=0.1*1+0.9*3=2.8

Expectation[importance(D)]=(prob(D=1|A=1,B=2,C=1)*importance(D=1)+(prob(D=2|A=1,B=2,C=1)*importance(D=2)=0.7*2+0.3*4=2.6

Hence, the expected feature importance (also referred to as relevance)from the answer of question E is determined to be higher than that ofquestion D, even though question D has generally higher featureimportances for its answers. The feature recommendation module cantherefore select question E as the next question to be presented toSubject X.

When selecting the next best feature to be presented to a subject, thefeature recommendation module 125 may be further configured to excludeone or more candidate features from consideration, if the candidatefeatures have a high co-variance with a feature that has already beenpresented to the subject. The co-variance of different features may bedetermined based on the training data, and may be stored in theassessment model constructed by the training module. If a candidatefeature has a high co-variance with a previously presented feature, thecandidate feature may add relatively little additional predictiveutility, and may hence be omitted from future presentation to thesubject in order to optimize the efficiency of the assessment procedure.

The prediction module 120 may interact with the person participating inthe assessment procedure (e.g., a subject or the subject's caretaker)with a user interface 130. The user interface may be provided with auser interface, such as a display of any computing device that canenable the user to access the prediction module, such as a personalcomputer, a tablet, or a smartphone. The computing device may comprise aprocessor that comprises instructions for providing the user interface,for example in the form of a mobile application. The user interface canbe configured to display instructions from the prediction module to theuser, and/or receive input from the user with an input method providedby the computing device. Thus, the user can participate in theassessment procedure as described herein by interacting with theprediction module with the user interface, for example by providinganswers (feature values) in response to questions (features) presentedby the prediction module. The user interface may be configured toadminister the assessment procedure in real-time, such that the useranswers one question at a time and the prediction module can select thenext best question to ask based on recommendations made by the featurerecommendation module. Alternatively or in combination, the userinterface may be configured to receive a complete set of new data from auser, for example by allowing a user to upload a complete set of featurevalues corresponding to a set of features.

As described herein, the features of interest relevant to identifyingone or more developmental disorders may be evaluated in a subject inmany ways. For example, the subject or caretaker or clinician may beasked a series of questions designed to assess the extent to which thefeatures of interest are present in the subject. The answers providedcan then represent the corresponding feature values of the subject. Theuser interface may be configured to present a series of questions to thesubject (or any person participating in the assessment procedure onbehalf of the subject), which may be dynamically selected from a set ofcandidate questions as described herein. Such a question-and-answerbased assessment procedure can be administered entirely by a machine,and can hence provide a very quick prediction of the subject'sdevelopmental disorder(s).

Alternatively or in combination, features of interest in a subject maybe evaluated with observation of the subject's behaviors, for examplewith videos of the subject. The user interface may be configured toallow a subject or the subject's caretaker to record or upload one ormore videos of the subject. The video footage may be subsequentlyanalyzed by qualified personnel to determine the subject's featurevalues for features of interest. Alternatively or in combination, videoanalysis for the determination of feature values may be performed by amachine. For example, the video analysis may comprise detecting objects(e.g., subject, subject's spatial position, face, eyes, mouth, hands,limbs, fingers, toes, feet, etc.), followed by tracking the movement ofthe objects. The video analysis may infer the gender of the subject,and/or the proficiency of spoken language(s) of the subject. The videoanalysis may identify faces globally, or specific landmarks on the facesuch as the nose, eyes, lips and mouth to infer facial expressions andtrack these expressions over time. The video analysis may detect eyes,limbs, fingers, toes, hands, feet, and track their movements over timeto infer behaviors. In some cases, the analysis may further infer theintention of the behaviors, for example, a child being upset by noise orloud music, engaging in self-harming behaviors, imitating anotherperson's actions, etc. The sounds and/or voices recorded in the videofiles may also be analyzed. The analysis may infer a context of thesubject's behavior. The sound/voice analysis may infer a feeling of thesubject. The analysis of a video of a subject, performed by a humanand/or by a machine, can yield feature values for the features ofinterest, which can then be encoded appropriately for input into theprediction module. A prediction of the subject's developmental disordermay then be generated based on a fitting of the subject's feature valuesto the assessment model constructed using training data.

Alternatively or in combination, features of interest in a subject maybe evaluated through structured interactions with the subject. Forexample, the subject may be asked to play a game such as a computergame, and the performance of the subject on the game may be used toevaluate one or more features of the subject. The subject may bepresented with one or more stimuli (e.g., visual stimuli presented tothe subject via a display), and the response of the subject to thestimuli may be used to evaluate the subject's features. The subject maybe asked to perform a certain task (e.g., subject may be asked to popbubbles with his or her fingers), and the response of the subject to therequest or the ability of the subject to carry out the requested taskmay be used to evaluate to the subject's features.

The methods and apparatus described herein can be configured in manyways to determine the next most predictive or relevant question. Atleast a portion of the software instructions as described herein can beconfigured to run locally on a local device so as to provide the userinterface and present questions and receive answers to the questions.The local device can be configured with software instructions of anapplication program interface (API) to query a remote server for themost predictive next question. The API can return an identified questionbased on the feature importance as described herein, for example.Alternatively or in combination, the local processor can be configuredwith instructions to determine the most predictive next question inresponse to previous answers. For example, the prediction module 120 maycomprise software instructions of a remote server, or softwareinstructions of a local processor, and combinations thereof.Alternatively or in combination, the feature recommendation module 125may comprise software instructions of a remote server, or softwareinstructions of a local processor, and combinations thereof, configuredto determine the most predictive next question, for example. Theexemplary operational flow 600 of method of determining an expectedfeature importance determination algorithm 127 as performed by a featurerecommendation module 125 described herein can be performed with one ormore processors as described herein, for example.

FIG. 7 illustrates a method 700 of administering an assessment procedureas described herein. The method 700 may be performed with a userinterface provided on a computing device, the computing devicecomprising a display and a user interface for receiving user input inresponse to the instructions provided on the display. The userparticipating in the assessment procedure may be the subject himself, oranother person participating in the procedure on behalf of the subject,such as the subject's caretaker. At step 705, an N^(th) question relatedan N^(th) feature can be presented to the user with the display. At step710, the subject's answer containing the corresponding N^(th) featurevalue can be received. At step 715, the dataset for the subject at handcan be updated to include N^(th) the feature value provided for thesubject. At step 720, the updated dataset can be fitted to an assessmentmodel to generate a predicted classification. Step 720 may be performedby a prediction module, as described herein. At step 725, a check can beperformed to determine whether the fitting of the data can generate aprediction of a specific developmental disorder (e.g., autism, ADHD,etc.) sufficient confidence (e.g., within at least a 90% confidenceinterval). If so, as shown at step 730, the predicted developmentaldisorder can be displayed to the user. If not, in step 735, a check canbe performed to determine whether there are any additional features thatcan be queried. If yes, as shown at step 740, the feature recommendationmodule may select the next feature to be presented to the user, andsteps 705-725 may be repeated until a final prediction (e.g., a specificdevelopmental disorder or “no diagnosis”) can be displayed to thesubject. If no additional features can be presented to the subject, “nodiagnosis” may be displayed to the subject, as shown at step 745.

Although the above steps show an exemplary a method 700 of administeringan assessment procedure, a person of ordinary skill in the art willrecognize many variations based on the teachings described herein. Thesteps may be completed in a different order. Steps may be added ordeleted. Some of the steps may comprise sub-steps of other steps. Manyof the steps may be repeated as often as desired by the user.

The present disclosure provides computer control systems that areprogrammed to implement methods of the disclosure. FIG. 8 shows acomputer system 801 suitable for incorporation with the methods andapparatus described herein. The computer system 801 can process variousaspects of information of the present disclosure, such as, for example,questions and answers, responses, statistical analyses. The computersystem 801 can be an electronic device of a user or a computer systemthat is remotely located with respect to the electronic device. Theelectronic device can be a mobile electronic device.

The computer system 801 includes a central processing unit (CPU, also“processor” and “computer processor” herein) 805, which can be a singlecore or multi core processor, or a plurality of processors for parallelprocessing. The computer system 801 also includes memory or memorylocation 810 (e.g., random-access memory, read-only memory, flashmemory), electronic storage unit 815 (e.g., hard disk), communicationinterface 820 (e.g., network adapter) for communicating with one or moreother systems, and peripheral devices 825, such as cache, other memory,data storage and/or electronic display adapters. The memory 810, storageunit 815, interface 820 and peripheral devices 825 are in communicationwith the CPU 805 through a communication bus (solid lines), such as amotherboard. The storage unit 815 can be a data storage unit (or datarepository) for storing data. The computer system 801 can be operativelycoupled to a computer network (“network”) 830 with the aid of thecommunication interface 820. The network 830 can be the Internet, aninternet and/or extranet, or an intranet and/or extranet that is incommunication with the Internet. The network 830 in some cases is atelecommunication and/or data network. The network 830 can include oneor more computer servers, which can enable distributed computing, suchas cloud computing. The network 830, in some cases with the aid of thecomputer system 801, can implement a peer-to-peer network, which mayenable devices coupled to the computer system 801 to behave as a clientor a server.

The CPU 805 can execute a sequence of machine-readable instructions,which can be embodied in a program or software. The instructions may bestored in a memory location, such as the memory 810. The instructionscan be directed to the CPU 805, which can subsequently program orotherwise configure the CPU 805 to implement methods of the presentdisclosure. Examples of operations performed by the CPU 805 can includefetch, decode, execute, and writeback.

The CPU 805 can be part of a circuit, such as an integrated circuit. Oneor more other components of the system 801 can be included in thecircuit. In some cases, the circuit is an application specificintegrated circuit (ASIC).

The storage unit 815 can store files, such as drivers, libraries andsaved programs. The storage unit 815 can store user data, e.g., userpreferences and user programs. The computer system 801 in some cases caninclude one or more additional data storage units that are external tothe computer system 801, such as located on a remote server that is incommunication with the computer system 801 through an intranet or theInternet.

The computer system 801 can communicate with one or more remote computersystems through the network 830. For instance, the computer system 801can communicate with a remote computer system of a user (e.g., aparent). Examples of remote computer systems and mobile communicationdevices include personal computers (e.g., portable PC), slate or tabletPC's (e.g., Apple® iPad, Samsung® Galaxy Tab), telephones, Smart phones(e.g., Apple® iPhone, Android-enabled device, Blackberry®), or personaldigital assistants. The user can access the computer system 801 with thenetwork 830.

Methods as described herein can be implemented by way of machine (e.g.,computer processor) executable code stored on an electronic storagelocation of the computer system 801, such as, for example, on the memory810 or electronic storage unit 815. The machine executable or machinereadable code can be provided in the form of software. During use, thecode can be executed by the processor 805. In some cases, the code canbe retrieved from the storage unit 815 and stored on the memory 810 forready access by the processor 805. In some situations, the electronicstorage unit 815 can be precluded, and machine-executable instructionsare stored on memory 810.

The code can be pre-compiled and configured for use with a machine havea processer adapted to execute the code, or can be compiled duringruntime. The code can be supplied in a programming language that can beselected to enable the code to execute in a pre-compiled or as-compiledfashion.

Aspects of the systems and methods provided herein, such as the computersystem 801, can be embodied in programming. Various aspects of thetechnology may be thought of as “products” or “articles of manufacture”typically in the form of machine (or processor) executable code and/orassociated data that is carried on or embodied in a type of machinereadable medium. Machine-executable code can be stored on an electronicstorage unit, such memory (e.g., read-only memory, random-access memory,flash memory) or a hard disk. “Storage” type media can include any orall of the tangible memory of the computers, processors or the like, orassociated modules thereof, such as various semiconductor memories, tapedrives, disk drives and the like, which may provide non-transitorystorage at any time for the software programming. All or portions of thesoftware may at times be communicated through the Internet or variousother telecommunication networks. Such communications, for example, mayenable loading of the software from one computer or processor intoanother, for example, from a management server or host computer into thecomputer platform of an application server. Thus, another type of mediathat may bear the software elements includes optical, electrical andelectromagnetic waves, such as used across physical interfaces betweenlocal devices, through wired and optical landline networks and overvarious air-links. The physical elements that carry such waves, such aswired or wireless links, optical links or the like, also may beconsidered as media bearing the software. As used herein, unlessrestricted to non-transitory, tangible “storage” media, terms such ascomputer or machine “readable medium” refer to any medium thatparticipates in providing instructions to a processor for execution.

Hence, a machine readable medium, such as computer-executable code, maytake many forms, including but not limited to, a tangible storagemedium, a carrier wave medium or physical transmission medium.Non-volatile storage media include, for example, optical or magneticdisks, such as any of the storage devices in any computer(s) or thelike, such as may be used to implement the databases, etc. shown in thedrawings. Volatile storage media include dynamic memory, such as mainmemory of such a computer platform. Tangible transmission media includecoaxial cables; copper wire and fiber optics, including the wires thatcomprise a bus within a computer system. Carrier-wave transmission mediamay take the form of electric or electromagnetic signals, or acoustic orlight waves such as those generated during radio frequency (RF) andinfrared (IR) data communications. Common forms of computer-readablemedia therefore include for example: a floppy disk, a flexible disk,hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD orDVD-ROM, any other optical medium, punch cards paper tape, any otherphysical storage medium with patterns of holes, a RAM, a ROM, a PROM andEPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wavetransporting data or instructions, cables or links transporting such acarrier wave, or any other medium from which a computer may readprogramming code and/or data. Many of these forms of computer readablemedia may be involved in carrying one or more sequences of one or moreinstructions to a processor for execution.

The computer system 801 can include or be in communication with anelectronic display 835 that comprises a user interface (UI) 840 forproviding, for example, questions and answers, analysis results,recommendations. Examples of UI's include, without limitation, agraphical user interface (GUI) and web-based user interface.

Methods and systems of the present disclosure can be implemented by wayof one or more algorithms and with instructions provided with one ormore processors as disclosed herein. An algorithm can be implemented byway of software upon execution by the central processing unit 805. Thealgorithm can be, for example, random forest, graphical models, supportvector machine or other.

Although the above steps show a method of a system in accordance with anexample, a person of ordinary skill in the art will recognize manyvariations based on the teaching described herein. The steps may becompleted in a different order. Steps may be added or deleted. Some ofthe steps may comprise sub-steps. Many of the steps may be repeated asoften as if beneficial to the platform.

Each of the examples as described herein can be combined with one ormore other examples. Further, one or more components of one or moreexamples can be combined with other examples.

Experimental Data

A data processing module as described herein was built on Python 2.7,Anaconda Distribution. The training data used to construct and train theassessment model included data generated by the Autism Genetic ResourceExchange (AGRE), which performed in-home assessments to collect ADI-Rand ADOS data from parents and children in their homes. ADI-R comprisesa parent interview presenting a total of 93 questions, and yields adiagnosis of autism or no autism. ADOS comprises a semi-structuredinterview of a child that yields a diagnosis of autism, ASD, or nodiagnosis, wherein a child is administered one of four possible modulesbased on language level, each module comprising about 30 questions. Thedata included clinical diagnoses of the children derived from theassessments; if a single child had discrepant ADI-R versus ADOSdiagnoses, a licensed clinical psychologist assigned a consensusdiagnosis for the dataset for the child in question. The training dataincluded a total of 3,449 data points, with 3,315 cases (autism or ASD)and 134 controls (non-spectrum). The features evaluated in the trainingdata targeted 3 key domains: language, social communication, andrepetitive behaviors.

A boosted Random Forest classifier was used to build the assessmentmodel as described herein. Prior to training the assessment model on thetraining data, the training data was pre-processed to standardize thedata, and re-encode categorical features in a one-hot representation asdescribed herein. Since the training data was skewed towards individualswith autism or ASD, sample weighting was applied to attribute up to 50times higher significance to data from non-spectrum individuals comparedto data from autistic/ASD individuals. The assessment model was trainediteratively with boosting, updating the weighting of data points aftereach iteration to increase the significance attributed to data pointsthat were misclassified, and retraining with the updated significances.

The trained model was validated using Stratified k-fold cross validationwith k=5. The cross-validation yielded an accuracy of about 93-96%,wherein the accuracy is defined as the percentage of subjects correctlyclassified using the model in a binary classification task(autism/non-spectrum). Since the training data contained a sample bias,a confusion matrix was calculated to determine how often the modelconfused one class (autism or non-spectrum) with another. The percentageof correctly classified autism individuals was about 95%, while thepercentage of correctly classified non-spectrum individuals was about76%. It should be noted, however, that the model may be adjusted to moreclosely fit one class versus another, in which case the percentage ofcorrect classifications for each class can change. FIG. 9 shows receiveroperating characteristic (ROC) curves mapping sensitivity versusfall-out for an exemplary assessment model as described herein. The truepositive rate (sensitivity) for the diagnosis of autism is mapped on they-axis, as a function of the false positive rate (fall-out) fordiagnosis mapped on the x-axis. Each of the three curves, labeled “Fold#0”, “Fold #1”, and “Fold #2”, corresponds to a different “fold” of thecross-validation procedure, wherein for each fold, a portion of thetraining data was fitted to the assessment model while varying theprediction confidence threshold necessary to classify a dataset as“autistic”. As desired or appropriate, the model may be adjusted toincrease the sensitivity in exchange for some increase in fall-out, orto decrease the sensitivity in return for a decrease in fall-out, asaccording to the ROC curves of the model.

The feature recommendation module was configured as described herein,wherein the expected feature importance of each question was computed,and candidate questions ranked in order of computed importance withcalls to a server with an application program interface (API). Thefeature recommendation module's ability to recommend informativequestions was evaluated by determining the correlation between aquestion's recommendation score with the increase in prediction accuracygained from answering the recommended question. The following steps wereperformed to compute the correlation metric: (1) the data was split upinto folds for cross-validation; (2) already answered questions wererandomly removed from the validation set; (3) expected featureimportance (question recommendation/score) was generated for eachquestion; (4) one of the questions removed in step 2 was revealed, andthe relative improvement in the subsequent prediction accuracy wasmeasured; and (5) the correlation between the relative improvement andthe expected feature importance was computed. The calculated Pearsoncorrelation coefficient ranged between 0.2 and 0.3, indicating amoderate degree of correlation between the expected feature importancescore and the relative improvement. FIG. 10 is a scatter plot showingthe correlation between the expected feature importance (“ExpectedInformativitiy Score”) and the relative improvement (“RelativeClassification Improvement”) for each question. The plot shows amoderate linear relationship between the two variables, demonstratingthe feature recommendation module is indeed able to recommend questionsthat would increase the prediction accuracy.

The length of time to produce an output using the developed predictionmodule and the feature recommendation model was measured. The predictionmodule took about 46 ms to make a prediction of an individual's risk ofautism. The feature recommendation module took about 41 ms to generationquestion recommendations for an individual. Although these measurementswere made with calls to a server through an API, the computations can beperformed locally, for example.

While the assessment model of the data processing module described withrespect to FIGS. 9-10 was constructed and trained to classify subjectsas having autism or no autism, a similar approach may be used to buildan assessment model that can classify a subject as having one or more ofa plurality of developmental disorders, as described herein.

In another aspect, the methods and apparatus disclosed herein canidentify a subject as belonging to one of three categories: having adevelopmental condition, being developmentally normal or typical, orinconclusive or requiring additional evaluation to determine whether thesubject has the developmental condition. The developmental condition canbe a developmental disorder or a developmental advancement. The additionof the third category, namely the inconclusive determination, results inimproved performance and better accuracy of the categorical evaluationscorresponding to the presence or absence of a developmental condition.

FIG. 11 is an exemplary operational flow of an evaluation moduleidentifying a subject as belonging to one of three categories. As shownin FIG. 11, a method 1100 is provided for evaluating at least onebehavioral developmental condition of a subject. The evaluation modulereceives diagnostic data of the subject related to the behavioraldevelopmental at 1110, evaluates the diagnostic data at 1120 using aselected subset of a plurality of machine learning assessment models andprovides categorical determinations for the subject at 1130. Thecategorical determination can be inconclusive, or can indicate thepresence or absence of the behavioral developmental condition.

FIG. 12 is an exemplary operational flow of a model training module asdescribed herein. As shown in FIG. 12, a method 1200 is provided forusing machine learning to train an assessment model and tune itsconfiguration parameters optimally. Multiple machine learning predictivemodels can be trained and tuned using the method 1200, each usingdatasets prepared offline and comprising a representative sample of astandardized clinical instrument such as ADI-R, ADOS, or SRS. Models canalso be trained using datasets comprising data other than clinicalinstruments, such as demographic data. The model training modulepre-processes diagnostic data from a plurality of subjects using machinelearning techniques at 1210. Datasets can be pre-processed usingwell-established machine learning techniques such as data cleaning,filtering, aggregation, imputation, normalization, and other machinelearning techniques as known in the art.

The model training module extracts and encodes machine learning featuresfrom the pre-processed diagnostic data at 1220. Columns comprising thedatasets can be mapped into machine learning features using featureencoding techniques such as, for example, one-hot encoding, severityencoding, presence-of-behavior encoding or any other feature encodingtechnique as known in the art. Some of these techniques are novel innature and not commonly used in machine learning applications, but theyare advantageous in the present application because of the nature of theproblem at hand, specifically because of the discrepancy between thesetting where clinical data is collected and the intended setting wherethe model will be applied.

Presence of behavior encoding in particular is advantageous for theproblem at hand especially, since the machine learning training data iscomprised of clinical questionnaires filled by psycho-metricians havingobserved subjects for multiple hours. The answer codes they fill in cancorrespond to subtle levels of severity or differences in behavioralpatterns that may only become apparent throughout the long period ofobservation. This data is then used to train models destined to beapplied in a setting where only a few minutes of subject observation isavailable. Hence the subtleties in behavioral patterns are expected tobe less often noticeable. Presence of behavioral encoding as describedherein mitigates this problem by abstracting away the subtle differencesbetween the answer choices and extracting data from the questionnairesonly at the level of granularity that is expected to be reliablyattained in the application setting.

The model training module processes the encoded machine learningfeatures at 1230. In an exemplary embodiment, questionnaire answers canbe encoded into machine learning features, after which, a sample weightcan be computed and assigned to every sample of diagnostic data in adataset, each sample corresponding to each subject having diagnosticdata. Samples can be grouped according to subject-specific dimensionsand sample weights can be computed and assigned to balance one group ofsamples against every other group of samples to mirror the expecteddistribution of subjects in an intended setting. For example, sampleswith positive classification labels might be balanced against those withnegative classification labels. Alternatively or additionally, samplesin each of multiple age group bins can be made to amount to an equaltotal weight. Additional sample balancing dimensions can be used such asgender, geographic region, sub-classification within the positive ornegative class, or any other suitable dimension.

The process of sample-weight adjustment might be further refined tomirror the expected distribution of subjects in the intended applicationsetting. This can allow the trained models to be adapted to variousspecific application settings. For example, a model can be trained foruse specifically as a level two screening tool by adjusting the sampleweights in the training dataset to reflect the expected prevalence ratesof diagnostic conditions in a level two diagnostic clinic. Anothervariant of the same screener can be trained for use as a general publicscreening tool, again by adjusting the weights of training samples toreflect and expected population of mostly neuro-typical subjects and aminority of positive samples with prevalence rates to match those in thegeneral population to mirror an expected distribution of subjects in anintended application setting.

The model training module selects a subset of the processed machinelearning features at 1240. In an exemplary embodiment, with the trainingsamples weighted accordingly, and all potential machine learningfeatures encoded appropriately, feature selection can take place using amachine learning process generally known as bootstrapping, wheremultiple iterations of model training can be run, each using a randomsubsample of the training data available. After each run, a tally can beupdated with the features the training process deemed necessary toinclude in the model. This list can be expected to vary from run to run,since the random data subsets used in training might contain apparentpatterns that are incidental to the choice of data samples and notreflective of real life patterns for the problem at hand. Repeating thisprocess multiple times can allow for the incidental patterns to cancelout, revealing the features that are reflective of patterns that can beexpected to generalize well outside the training dataset and into thereal world. The top features of the bootstrapping runs can then beselected and used exclusively for training the final model, which istrained using the entire training dataset, and saved for laterapplication.

Several models can be trained instead of one model, in order tospecialize the models over a demographic dimension in situations wherethe dimension is expected to affect the choice of useful features. Forexample, multiple questionnaire-based models can be built, each for aspecific age group, since the best questions to ask of a subject areexpected to be different for each age group. In this case, only theright model for each subject is loaded at application time.

The model training module evaluates each model at 1250. In particular,each model can be evaluated for performance, for example, as determinedby sensitivity and specificity for a pre-determined inclusion rate. Inan exemplary embodiment, using a held-out dataset that was not usedduring the model training phase, the models can be evaluated forperformance, in terms of inclusion rate, sensitivity, and specificity.

The model training module tunes each model at 1260. More specifically,to assess the performance of the models in different tuning settings,the tuning parameters of each model can be changed in iterativeincrements and the same metrics can be computed over the same held-outset in every iteration. The optimal settings can then be locked in andthe corresponding models saved. Tuning parameters can include, forexample, the number of trees in a boosted decision tree model, themaximum depth of every tree, the learning rate, the threshold ofpositive determination score, the range of output deemed inconclusive,and any other tuning parameter as known in the art.

In a preferable embodiment, the parameter tuning process of 1260 cancomprise a brute-force grid search, an optimized gradient descent orsimulated annealing, or any other space exploration algorithm as knownin the art. The models being tuned can undergo separate, independenttuning runs, or alternatively the models can be tuned in an ensemblefashion, with every parameter of every model explored in combination, inorder to arrive at the optimal overall set of parameters at 1270 tomaximize the benefit of using all the models in an ensemble.

Moreover, in yet another aspect, tuning the inconclusive range of eachpredictive model can be augmented with an external condition, determinedby a business need rather than a performance metric. For example, it canbe deemed necessary for a particular classifier to have an inclusionrate of no less than 70%. In other words, the classifier would beexpected to provide an evaluation indicating either the presence or theabsence of a developmental condition for at least 70% of the subjectsbeing classified, yielding an inconclusive determination for less than30% of the subjects. Accordingly, the corresponding tuning process forthe inconclusive output range would have to be limited to only theranges where this condition is met.

The models are tunable based on the context of the application. Thepredictive model can be configured to output a diagnosis having aparticular degree of certainty that can be adjusted based on tuning ofthe inconclusive range.

In addition, tuning of the inconclusive range can be exposed outside theoffline machine learning phase. More specifically, tuning of theinconclusive range can be a configurable parameter accessible to agentsoperating the models after deployment. In this way, it is possible foran operator to dial the overall system up or down along the tradeoffbetween more inclusion and more accuracy. To support this case, multipleoptimal inconclusive ranges might be explored and stored during themodel training phase, each with its corresponding inclusion rate. Theagent can then affect that change by selecting an optimal point from amenu of previously determined optimal settings.

FIG. 13 is another exemplary operational flow of an evaluation module asdescribed herein. As shown in FIG. 13, a method 1300 is provided foroutputting a conclusive prediction at 1355 indicating the presence orabsence of a developmental condition, or an inconclusive determinationof “No diagnosis” at 1365.

The evaluation module as depicted in FIG. 13 receives new data such asdiagnostic data from or associated with a subject to be evaluated ashaving or not having a developmental condition at 1310. Multiple savedassessment models that have been trained, tuned, and optimized asdepicted in FIG. 12 and as described herein can be loaded at 1320.Diagnostic data can be fit to these initial assessment models andoutputs can be collected at 1330. The evaluation module can combine theinitial assessment model outputs at 1340 to generate a predicted initialclassification of the subject. If the evaluation module determines thatthe initial prediction is conclusive at 1350, it can output a conclusivedetermination indicating either the presence or absence of thedevelopmental condition in the subject. If the evaluation moduledetermines that the initial prediction is inconclusive at 1350, it canthen proceed to determine whether additional or more sophisticatedassessment models are available and applicable at 1360. If no additionalassessment models are available or applicable, the evaluation moduleoutputs an inconclusive determination of “No diagnosis.” If however, theevaluation module determines that additional or more sophisticatedassessment models are available and applicable, it can proceed to obtainadditional diagnostic data from or associated with the subject at 1370.Next, the evaluation module can load the additional or moresophisticated assessment models at 1380 and can repeat the process offitting data to the models, only this time, the additional data obtainedat 1370 is fitted to the additional assessment models loaded at 1380 toproduce new model outputs, which are then evaluated at 1350 for aconclusive prediction. This process as depicted by the loop comprisingsteps 1350, 1355, 1360, 1365, 1370, 1380 and back to 1330 and 1340 canbe repeated until either a conclusive prediction is output at 1355, orif no more applicable classification models are available to use, aninconclusive determination of “No diagnosis” is output at 1365.

In particular, when data from a new subject is received as input at 1310in FIG. 13, each available model for preliminary determination is loadedat 1320 and run, outputting a numerical score at 1330. The scores canthen be combined using a combinatorial model.

FIG. 14 is an exemplary operational flow of the model output combiningstep depicted in FIG. 13. As shown in FIG. 14, a combiner module 1400can collect the outputs from multiple assessment models 1410, 1420,1430, and 1440, which are received by a model combinatory orcombinatorial model 1450. The combinatorial model can employ simplerule-based logic to combine the outputs, which can be numerical scores.Alternatively, the combinatorial model can use more sophisticatedcombinatorial techniques such as logistic regression, probabilisticmodeling, discriminative modeling, or any other combinatorial techniqueas known in the art. The combinatorial model can also rely on context todetermine the best way to combine the model outputs. For example, it canbe configured to trust the questionnaire-based model output only in acertain range, or to defer to the video-based model otherwise. Inanother case, it can use the questionnaire-based model output moresignificantly for younger subjects than older ones. In another case, itcan exclude the output of the video-based model for female subjects, butinclude the video-based model for male subjects.

The combinatorial model output score can then be subjected to thresholdsdetermined during the model training phase as described herein. Inparticular, as shown in FIG. 14, these thresholds are indicated by thedashed regions that partition the range of numerical scores 1460 intothree segments corresponding to a negative determination output 1470, aninconclusive determination output 1480, and a positive determinationoutput 1490. This effectively maps the combined numerical score to acategorical determination, or to an inconclusive determination if theoutput is within the predetermined inconclusive range.

In the case of an inconclusive output, the evaluation module candetermine that additional data should be obtained from the subject inorder to load and run additional models beyond the preliminary orinitial set of models. The additional models might be well suited todiscern a conclusive output in cases where the preliminary models mightnot. This outcome can be realized by training additional models that aremore sophisticated in nature, more demanding of detailed input data, ormore focused on the harder-to-classify cases to the exclusion of thestraightforward ones.

FIG. 15 shows an exemplary questionnaire screening algorithm configuredto provide only categorical determinations of a developmental conditionas described herein. In particular, the questionnaire screeningalgorithm depicted in FIG. 15 shows an alternating decision treeclassifier that outputs a determination indicating only the presence orthe absence of autism. The different shading depicts the totalpopulation of children who are autistic and not autistic and who areevaluated via the questionnaire. Also depicted are the results of theclassifier, showing the correctly and incorrectly diagnosed childrenpopulations for each of the two categorical determinations.

In contrast, FIG. 16 shows an exemplary Triton questionnaire screeningalgorithm configured to provide both categorical and inconclusivedeterminations as described herein. In particular, the Triton algorithmdepicted in FIG. 16 implements both age-appropriate questionnaires andage-specific models to yield specialized classifiers for each of twosubgroups (i.e. “3 years old & under” and “4+ year olds”) within arelevant age group (i.e. “children”). It is clear from this example thatthe categorical determinations indicating the presence and absence ofAutism in the two subgroups in FIG. 16 each have a higher accuracy whencompared with the categorical determinations in FIG. 15, as indicated bythe different shaded areas showing the correctly and incorrectlydiagnosed children populations for each of the two categoricaldeterminations. By providing a separate category for inconclusivedeterminations, the Triton algorithm of FIG. 16 is better able toisolate hard-to-screen cases that result in inaccurate categoricaldeterminations as seen in FIG. 15.

A comparison of the performance for various algorithms highlights theadvantages of the Triton algorithm, and in particular, the Tritonalgorithm having a context-dependent combination of questionnaire andvideo inputs. FIG. 17 shows a comparison of the performance for variousalgorithms in terms of a sensitivity-specificity tradeoff for allsamples in a clinical sample as described herein. As shown in FIG. 17,the best performance in terms of both sensitivity and specificity isobtained by the Triton algorithm configured for 70% coverage whencombined with the video combinator (i.e. context-dependent combinationof questionnaire and video inputs).

FIG. 18 shows a comparison of the performance for various algorithms interms of a sensitivity-specificity tradeoff for samples taken fromchildren under 4 as described herein. The Triton algorithm configuredfor 70% coverage when combined with the video combinator (i.e.context-dependent combination of questionnaire and video inputs) has thebest performance.

FIG. 19 shows a comparison of the performance for various algorithms interms of a sensitivity-specificity tradeoff for samples taken fromchildren 4 and over described herein. For the most part, the Tritonalgorithm configured for 70% coverage when combined with the videocombinator appears to have the best performance.

FIGS. 20-22, show the specificity for different algorithms at 75%-85%sensitivity range for all samples, for children under 4, and forchildren 4 and over. In all three cases, the Triton algorithm configuredfor 70% coverage when combined with the video combinator has the bestperformance, having 75% specificity for all samples, 90% specificity forchildren under 4, and 55% specificity for children 4 and over. Note thatthe Triton algorithm has the further advantage of flexibility. Forexample, tunable models are provided as described herein, wherein theinconclusive ratio or inclusion rate may be controlled or adjusted tocontrol the tradeoff between coverage and reliability. In addition, themodels described herein may be tuned to an application setting withrespect to expected prevalence rates or based on expected populationdistributions for a given application setting. Finally, support foradaptive retraining enables improved performance over time given thefeedback training loop of the method and system described herein.

A person of ordinary skill in the art can generate and obtain additionaldatasets and improve the sensitivity and specificity and confidenceinterval of the methods and apparatus disclosed herein to obtainimproved results without undue experimentation. Although thesemeasurements were performed with example datasets, the methods andapparatus can be configured with additional datasets as described hereinand the subject identified as at risk with a confidence interval of 80%in a clinical environment without undue experimentation. The sensitivityand specificity of 80% or more in a clinical environment can besimilarly obtained with the teachings provided herein by a person ofordinary skill in the art without undue experimentation, for examplewith additional datasets.

Additional datasets may be obtained from large archival datarepositories as described herein, such as the Autism Genetic ResourceExchange (AGRE), Boston Autism Consortium (AC), Simons Foundation,National Database for Autism Research, and the like. Alternatively or incombination, additional datasets may comprise mathematically simulateddata, generated based on archival data using various simulationalgorithms. Alternatively or in combination, additional datasets may beobtained via crowd-sourcing, wherein subjects self-administer theassessment procedure as described herein and contribute data from theirassessment. In addition to data from the self-administered assessment,subjects may also provide a clinical diagnosis obtained from a qualifiedclinician, so as to provide a standard of comparison for the assessmentprocedure.

In another aspect, a digital personalized medicine system as describedherein comprises digital devices with processors and associated softwareconfigured to: receive data to assess and diagnose a patient; captureinteraction and feedback data that identify relative levels of efficacy,compliance and response resulting from the therapeutic interventions;and perform data analysis, including at least one or machine learning,artificial intelligence, and statistical models to assess user data anduser profiles to further personalize, improve or assess efficacy of thetherapeutic interventions.

The assessment and diagnosis of the patient in the digital personalizedmedicine system can categorize a subject into one of three categories:having one or more developmental conditions, being developmentallynormal or typical, or inconclusive (i.e. requiring additional evaluationto determine whether the subject has any developmental conditions). Inparticular, a separate category can be provided for inconclusivedeterminations, which results in greater accuracy with respect tocategorical determinations indicating the presence or absence of adevelopmental condition. A developmental condition can be adevelopmental disorder or a developmental advancement. Moreover, themethods and apparatus disclosed herein are not limited to developmentalconditions, and may be applied to other cognitive functions, such asbehavioral, neurological or mental health conditions.

In some instances, the system can be configured to use digitaldiagnostics and digital therapeutics. Digital diagnostics and digitaltherapeutics can comprise a system or methods comprising collectingdigital information and processing and evaluating the provided data toimprove the medical, psychological, or physiological state of anindividual. The system and methods described herein can categorize asubject into one of three categories: having one or more developmentalconditions, being developmentally normal or typical, or inconclusive(i.e. requiring additional evaluation to determine whether the subjecthas any developmental conditions). In particular, a separate categorycan be provided for inconclusive determinations, which results ingreater accuracy with respect to categorical determinations indicatingthe presence or absence of a developmental condition. A developmentalcondition can be a developmental disorder or a developmentaladvancement. Moreover, the methods and apparatus disclosed herein arenot limited to developmental conditions, and may be applied to othercognitive functions, such as behavioral, neurological or mental healthconditions. In addition, a digital therapeutic system can apply softwarebased learning to evaluate user data, monitor and improve the diagnosesand therapeutic interventions provided by the system.

Digital diagnostics in the system can comprise of data and meta-datacollected from the patient, or a caregiver, or a party that isindependent of the individual being assessed. In some instances thecollected data can comprise monitoring behaviors, observations,judgements, or assessments may be made by a party other than theindividual. In further instances, the assessment can comprise an adultperforming an assessment or provide data for an assessment of a child orjuvenile.

Data sources can comprise either active or passive sources, in digitalformat via one or more digital devices such as mobile phones, videocapture, audio capture, activity monitors, or wearable digital monitors.Examples of active data collection comprise devices, systems or methodsfor tracking eye movements, recording body or appendage movement,monitoring sleep patterns, recording speech patterns. In some instances,the active sources can include audio feed data source such as speechpatterns, lexical/syntactic patterns (for example, size of vocabulary,correct/incorrect use of pronouns, correct/incorrect inflection andconjugation, use of grammatical structures such as active/passive voiceetc., and sentence flow), higher order linguistic patterns (for example,coherence, comprehension, conversational engagement, and curiosity).Active sources can also include touch-screen data source (for example,fine-motor function, dexterity, precision and frequency of pointing,precision and frequency of swipe movement, and focus/attention span).Video recording of subject's face during activity (for example,quality/quantity of eye fixations vs saccades, heat map of eye focus onthe screen, focus/attention span, variability of facial expression, andquality of response to emotional stimuli) can also be considered anactive source of data.

Passive data collection can comprise devices, systems, or methods forcollecting data from the user using recording or measurements derivedfrom mobile applications, toys with embed sensors or recording units. Insome instances, the passive source can include sensors embedded in smarttoys (for example, fine motor function, gross motor function,focus/attention span and problem solving skills) and wearable devices(for example, level of activity, quantity/quality of rest).

The data used in the diagnosis and treatment can come from a pluralityof sources, and may comprise a combination of passive and active datacollection gathered from one device such as a mobile device with whichthe user interacts, or other sources such as microbiome sampling andgenetic sampling of the subject.

The methods and apparatus disclosed herein are well suited for thediagnosis and digital therapeutic treatment of cognitive anddevelopmental disorders, mood and mental illness, and neurodegenerativediseases. Examples of cognitive and developmental disorders includespeech and learning disorders and other disorders as described herein.Examples of mood and mental illness disorders, which can effect childrenand adults, include behavioral disorders, mood disorders, depression,attention deficit hyperactivity disorder (“ADHD”), obsessive compulsivedisorder (“OCD”), schizophrenia, and substance-related disorders such aseating disorders and substance abuse. Examples of neurodegenerativediseases include age related cognitive decline, cognitive impairmentprogressing to Alzheimer's and senility, Parkinson's disease andHuntington's disease, and amyotrophic lateral sclerosis (“ALS”). Themethods and apparatus disclosed herein are capable of digitallydiagnosing and treating children and continuing treatment until thesubject becomes an adult, and can provide lifetime treatment based onpersonalized profiles.

The digital diagnosis and treatment as described herein is well suitedfor behavioral intervention coupled with biological or chemicaltherapeutic treatment. By gathering user interaction data as describedherein, therapies can be provided for combinations of behavioralintervention data pharmaceutical and biological treatments.

The mobile devices as described herein may comprise sensors to collectdata of the subject that can be used as part of the feedback loop so asto improve outcomes and decrease reliance on user input. The mobiledevice may comprise passive or active sensors as described herein tocollect data of the subject subsequent to treatment. The same mobiledevice or a second mobile device, such as an iPad™ or iPhone™ or similardevice, may comprise a software application that interacts with the userto tell the user what to do in improve treatment on a regular basis,e.g. day by day, hour by hour, etc. The user mobile device can beconfigured to send notifications to the user in response to treatmentprogress. The mobile device may comprise a drug delivery deviceconfigured to monitor deliver amounts of a therapeutic agent deliveredto the subject.

The methods and apparatus disclosed herein are well suited for treatmentof both parents and children, for example. Both a parent and a child canreceive separate treatments as described herein. For example,neurological condition of the parent can be monitored and treated, andthe developmental progress of the child monitored and treated.

The mobile device used to acquire data of the subject can be configuredin many ways and may combine a plurality of devices, for example. Forexample, since unusual sleep patterns may be related to autism, sleepdata acquired using the therapeutic apparatus described herein can beused as an additional input to the machine learning training process forautism classifiers used by the diagnostic apparatus described above. Themobile device may comprise a mobile wearable for sleep monitoring for achild, which can be provide as input for diagnosis and treatment and maycomprise a component of the feedback loop as described herein.

Many types of sensor, biosensors and data can be used to gather data ofthe subject and input into the diagnosis and treatment of the subject.For example, work in relation to embodiments suggests that microbiomedata can be useful for the diagnosis and treatment of autism. Themicrobiome data can be collected in many ways known to one of ordinaryskill in the art, and may comprise data selected from a stool sample,intestinal lavage, or other sample of the flora of the subject'sintestinal track. Genetic data can also be acquired an input into thediagnostic and therapeutic modules. The genetic data may comprise fullgenomic sequencing of the subject, of sequencing and identification ofspecific markers.

The diagnostic and therapeutic modules as disclosed herein can receivedata from a plurality of sources, such as data acquired from the groupconsisting of genetic data, floral data, a sleep sensor, a wearableanklet sleep monitor, a booty to monitor sleep, and eye tracking of thesubject. The eye tracking can be performed in many ways to determine thedirection and duration of gaze. The tracking can be done with glasses,helmets or other sensors for direction and duration of gaze. The datacan be collected during a visual session such as a video playback orvideo game, for example. This data can be acquired and provided to thetherapeutic module and diagnostic module as described herein before,during and after treatment, in order to initially diagnose the subject,determine treatment of the subject, modify treatment of the subject, andmonitor the subject subsequent to treatment.

The visual gaze, duration of gaze and facial expression information canbe acquired with methods and apparatus known to one of ordinary skill inthe art, and acquired as input into the diagnostic and therapeuticmodules. The data can be acquired with an app comprising softwareinstructions, which can be downloaded. For example, facial processinghas been described by Gloarai et al. “Autism and the development of faceprocessing”, Clinical Neuroscience Research 6 (2006) 145-160. An autismresearch group at Duke University has been conducting the Autism andbeyond research study with a software app downloaded onto mobile devicesas described on the web page at autismandbeyond.researchkit.duke.edu.Data from such devices is particularly well suited for combination inaccordance with the present disclosure. Facial recognition data and gazedata can be input into the diagnostic and therapeutic modules asdescribed herein.

The classifiers as disclosed herein are particularly well suited forcombination with this data to provide improved therapy and treatment.The data can be stratified and used with a feedback loop as describedherein. For example, the feedback data can be used in combination with adrug therapy to determine differential responses and identify respondersand non-responders. Alternatively or in combination, the feedback datacan be combined with non-drug therapy, such as behavioral therapy.

With regards to genetics, recent work suggests that some people may havegenes that make them more susceptible to Autism. The genetic compositionof the subject may render the subject more susceptible to environmentalinfluences, which can cause symptoms and may influence the severity ofsymptoms. The environmental influence may comprise an insult from atoxin, virus or other substance, for example. Without being bound by anyparticular theory, this may result in mechanisms that change theregulation of expression genes. The change in expression of genes may berelated to change in gastro-intestinal (“GI”) flora, and these changesin flora may affect symptoms related to Autism. Alternatively or incombination, an insult to the intestinal microbiome may result in achange in the microbiome of the subject, resulting in the subject havingless than ideal homeostasis, which may affect associated symptomsrelated to Autism. The inventors note that preliminary studies with B.fragilis conducted by Sarkis K. Mazmanian and others, suggest changes inthis micro-organism can be related to autism and the development ofautisms. (See also, “Gut Bacteria May Play a Role in Autism” by MelindaWenner Moyer, Scientific American, Sep. 1, 2014)

The digital diagnostic uses the data collected by the system about thepatient, which may include complimentary diagnostic data capturedoutside the digital diagnostic, with analysis from tools such as machinelearning, artificial intelligence, and statistical modeling to assess ordiagnose the patient's condition. The digital diagnostic can alsoprovide assessment of a patient's change in state or performance,directly or indirectly via data and meta-data that can be analyzed andevaluated by tools such as machine learning, artificial intelligence,and statistical modeling to provide feedback into the system to improveor refine the diagnoses and potential therapeutic interventions.

Analysis of the data comprising digital diagnostic, digitaltherapeutics, and corresponding responses, or lack thereof, from thetherapeutic interventions can lead to the identification of noveldiagnoses for patients and novel therapeutic regimens for both patentsand caregivers.

Types of data collected and utilized by the system can include patientand caregiver video, audio, responses to questions or activities, andactive or passive data streams from user interaction with activities,games or software features of the system, for example. Such data canalso represent patient or caregiver interaction with the system, forexample, when performing recommended activities. Specific examplesinclude data from a user's interaction with the system's device ormobile app that captures aspects of the user's behaviors, profile,activities, interactions with the software system, interactions withgames, frequency of use, session time, options or features selected, andcontent and activity preferences. Data may also include streams fromvarious third party devices such as activity monitors, games orinteractive content.

Digital therapeutics as described herein can comprise of instructions,feedback, activities or interactions provided to the patient orcaregiver by the system. Examples include suggested behaviors,activities, games or interactive sessions with system software and/orthird party devices (for example, the Internet of Things “IoT” enabledtherapeutic devices as understood by one of ordinary skill in the art).

FIG. 23A illustrates a system diagram for a digital personalizedmedicine platform 2300 for providing diagnosis and therapy related tobehavioral, neurological or mental health disorders. The platform 2300can provide diagnosis and treatment of pediatric cognitive andbehavioral conditions associated with developmental delays, for example.A user digital device 2310—for example, a mobile device such as a smartphone, an activity monitor, or a wearable digital monitor—records dataand metadata related to a patient. Data may be collected based oninteractions of the patient with the device, as well as based oninteractions with caregivers and health care professionals. The data maybe collected actively, such as by administering tests, recording speechand/or video, and recording responses to diagnostic questions. The datamay also be collected passively, such as by monitoring online behaviorof patients and caregivers, such as recording questions asked and topicsinvestigated relating to a diagnosed developmental disorder.

The digital device 2310 is connected to a computer network 2320,allowing it to share data with and receive data from connectedcomputers. In particular, the device can communicate with personalizedmedical system 2330, which comprises a server configured to communicatewith digital device 2310 over the computer network 2320. Personalizedmedical system 2330 comprises a diagnosis module 2332 to provide initialand incremental diagnosis of a patient's developmental status, as wellas a therapeutic module 2334 to provide personalized therapyrecommendations in response to the diagnoses of diagnosis module 2332.

Each of diagnosis modules 2332 and 2334 communicate with the userdigital device 2310 during a course of treatment. The diagnosis moduleprovides diagnostic tests to and receives diagnostic feedback from thedigital device 2310, and uses the feedback to determine a diagnosis of apatient. An initial diagnosis may be based on a comprehensive set oftests and questions, for example, while incremental updates may be madeto a diagnosis using smaller data samples. For example, the diagnosticmodule may diagnose autism-related speech delay based on questions askedto the caregiver and tests administered to the patient such asvocabulary or verbal communication tests. The diagnosis may indicate anumber of months or years delay in speech abilities. Later tests may beadministered and questions asked to update this diagnosis, for exampleshowing a smaller or larger degree of delay.

The diagnosis module communicates its diagnosis to the digital device2310, as well as to therapy module 2334, which uses the diagnosis tosuggest therapies to be performed to treat any diagnosed symptoms. Thetherapy module 2334 sends its recommended therapies to the digitaldevice 2310, including instructions for the patient and caregivers toperform the therapies recommended over a given time frame. Afterperforming the therapies over the given time frame, the caregivers orpatient can indicate completion of the recommended therapies, and areport can be sent from the digital device 2310 to the therapy module2334. The therapy module 2334 can then indicate to the diagnosis module2332 that the latest round of therapy is finished, and that a newdiagnosis is needed. The diagnostic module 2332 can then provide newdiagnostic tests and questions to the digital device 2310, as well astake input from the therapy module of any data provided as part oftherapy, such as recordings of learning sessions or browsing history ofcaregivers or patients related to the therapy or diagnosed condition.The diagnostic module 2332 then provides an updated diagnosis to repeatthe process and provide a next step of therapy.

Information related to diagnosis and therapy can also be provided frompersonalized medical system 2330 to a third-party system 2340, such as acomputer system of a health care professional. The health careprofessional or other third party can be alerted to significantdeviations from a therapy schedule, including whether a patient isfalling behind an expected schedule or is improving faster thanpredicted. Appropriate further action can then be taken by the thirdparty based on this provided information.

FIG. 23B illustrates a detailed diagram of diagnosis module 2332. Thediagnosis module 2332 comprises a test administration module 2342 thatgenerates tests and corresponding instructions for administration to asubject. The diagnosis module 2332 also comprises a subject datareceiving module 2344 in which subject data are received, such as testresults; caregiver feedback; meta-data from patient and caregiverinteractions with the system; and video, audio, and gaming interactionswith the system, for example. A subject assessment module 2346 generatesa diagnosis of the subject based on the data from subject data receivingmodule 2344, as well as past diagnoses of the subject and of similarsubjects. A machine learning module 2348 assesses the relativesensitivity of each input to the diagnosis to determine which types ofmeasurement provide the most information regarding a patient'sdiagnosis. These results can be used by test administration module 2342to provide tests which most efficiently inform diagnoses and by subjectassessment module 2346 to apply weights to diagnosis data in order toimprove diagnostic accuracy and consistency. Diagnostic data relating toeach treated patient are stored, for example in a database, to form alibrary of diagnostic data for pattern matching and machine learning. Alarge number of subject profiles can be simultaneously stored in such adatabase, for example 10,000 or more.

FIG. 23C illustrates a detailed diagram of therapy module 2334. Therapymodule 2334 comprises a therapy assessment module 2352 that scorestherapies based on their effectiveness. A previously suggested therapyis evaluated based on the diagnoses provided by the diagnostic moduleboth before and after the therapy, and a degree of improvement isdetermined. This degree of improvement is used to score theeffectiveness of the therapy. The therapy may have its effectivenesscorrelated with particular classes of diagnosis; for example, a therapymay be considered effective for subjects with one type of diagnosis butineffective for subjects with a second type of diagnosis. A therapymatching module 2354 is also provided that compares the diagnosis of thesubject from diagnosis module 2332 with a list of therapies to determinea set of therapies that have been determined by the therapy assessmentmodule 2352 to be most effective at treating diagnoses similar to thesubject's diagnosis. Therapy recommendation module 2356 then generates arecommended therapy comprising one or more of the therapies identifiedas promising by the therapy matching module 2354, and sends thatrecommendation to the subject with instructions for administration ofthe recommended therapies. Therapy tracking module 2358 then tracks theprogress of the recommended therapies, and determines when a newdiagnosis should be performed by diagnosis module 2332, or when a giventherapy should be continued and progress further monitored. Therapeuticdata relating to each patient treated are stored, for example in adatabase, to form a library of therapeutic data for pattern matching andmachine learning. A large number of subject profiles can besimultaneously stored in such a database, for example 10,000 or more.The therapeutic data can be correlated to the diagnostic data of thediagnostic module 2332 to allow a matching of effective therapies todiagnoses.

A therapy can comprise a digital therapy. A digital therapy can comprisea single or multiplicity of therapeutic activities or interventions thatcan be performed by the patient or caregiver. The digital therapeuticcan include prescribed interactions with third party devices such assensors, computers, medical devices and therapeutic delivery systems.Digital therapies can support an FDA approved medical claim, a set ofdiagnostic codes, or a single diagnostic code.

FIG. 24 illustrates a method 2400 for diagnosis and therapy to beprovided in a digital personalized medicine platform. The digitalpersonalized medicine platform communicates with a subject, which mayinclude a patient with one or more caregivers, to provide diagnoses andrecommend therapies.

In step 2410 the diagnosis module assesses the subject to determine adiagnosis, for example by applying diagnostic tests to the subject. Thediagnostic tests may be directed at determining a plurality of featuresand corresponding feature values for the subject. For example, the testsmay include a plurality of questions presented to a subject,observations of the subject, or tasks assigned to the subject. The testsmay also include indirect tests of the subject, such as feedback from acaregiver of patient performance versus specific behaviors and/ormilestones; meta-data from patient and caregiver interactions with thesystem; and video, audio, and gaming interactions with the system orwith third party tools that provide data on patient and caregiverbehavior and performance. For initial tests, a more comprehensivetesting regimen may be performed, aimed at generating an accurateinitial diagnosis. Later testing used to update prior diagnoses to trackprogress can involve less comprehensive testing and may, for example,rely more on indirect tests such as behavioral tracking andtherapy-related recordings and meta-data.

In step 2412, the diagnosis module receives new data from the subject.The new data can comprise an array of features and corresponding featurevalues for a particular subject. As described herein, the features maycomprise a plurality of questions presented to a subject, observationsof the subject, or tasks assigned to the subject. The feature values maycomprise input data from the subject corresponding to characteristics ofthe subject, such as answers of the subject to questions asked, orresponses of the subject. The feature values may also comprise recordedfeedback, meta-data, and system interaction data as described above.

In step 2414, the diagnosis module can load a previously savedassessment model from a local memory and/or a remote server configuredto store the model. Alternatively, if no assessment model exists for thepatient, a default model may be loaded, for example, based on one ormore initial diagnostic indications.

In step 2416, the new data is fitted to the assessment model to generatean updated assessment model. This assessment model may comprise aninitial diagnosis for a previously untreated subject, or an updateddiagnosis for a previously treated subject. The updated diagnosis caninclude a measurement of progress in one or more aspects of a condition,such as memory, attention and joint attention, cognition, behavioralresponse, emotional response, language use, language skill, frequency ofspecific behaviors, sleep, socialization, non-verbal communication, anddevelopmental milestones. The analysis of the data to determine progressand current diagnosis can include automated analysis such as questionscoring and voice-recognition for vocabulary and speech analysis. Theanalysis can also include human scoring by analysis reviewing video,audio, and text data.

In step 2418, the updated assessment model is provided to the therapymodule, which determines what progress has been made as a result of anypreviously recommended therapy. The therapy module scores the therapybased on the amount of progress in the assessment model, with largerprogress corresponding to a higher score, making a successful therapyand similar therapies more likely to be recommended to subjects withsimilar assessments in the future. The set of therapies available isthus updated to reflect a new assessment of effectiveness, as correlatedwith the subject's diagnosis.

In step 2420, a new therapy is recommended based on the assessmentmodel, the degree of success of the previous therapy, if any, and thescores assigned to a collection of candidate therapies based on previoususes of those therapies with the subject and other subjects with similarassessments. The recommended therapy is sent to the subject foradministration, along with instructions of a particular span of time toapply it. For example, a therapy might include a language drill to beperformed with the patient daily for one week, with each drill to berecorded in an audio file in a mobile device used by a caregiver or thepatient.

In step 2422, progress of the new therapy is monitored to determinewhether to extend a period of therapy. This monitoring may includeperiodic re-diagnoses, which may be performed by returning to step 2410.Alternatively, basic milestones may be recorded without a fullre-diagnosis, and progress may be compared to a predicted progressschedule generated by the therapy module. For example, if a therapy isunsuccessful initially, the therapy module may suggest repeating it oneor more times before either re-diagnosing and suggesting a new therapyor suggesting intervention by medical professionals.

FIG. 25 illustrates a flow diagram 2500 showing the handling ofsuspected or confirmed speech and language delay.

In step 2502 an initial assessment is determined by diagnosis module2532. The initial assessment can assess the patient's performance in oneor more domains, such as speech and language use, and assess a degreeand type of developmental delay along a number of axes, as disclosedherein. The assessment can further place the subject into one of aplurality of overall tracks of progress; for example, the subject can beassessed as verbal or nonverbal.

If the subject is determined to be non-verbal, as in step 2510, one ormore non-verbal therapies 2512 can be recommended by the therapy module2534, such as tasks related to making choices, paying attention totasks, or responding to a name or other words. Further suggestions ofuseful devices and products that may be helpful for progress may also beprovided, and all suggestions can be tailored to the subject's needs asindicated by the subject's diagnosis and progress reports.

While applying the recommended therapies, progress is monitored in step2514 to determine whether a diagnosis has improved at a predicted rate.

If improvement has been measured in step 2514, the system determineswhether the subject is still non-verbal in step 2516; if so, then thesystem returns to step 2510 and generates a new recommended therapy 2512to induce further improvements.

If no improvement is measured in step 2514, the system can recommendthat the therapy be repeated a predetermined number of times. The systemmay also recommend trying variations in therapy to try and get betterresults. If such repetitions and variations fail, the system canrecommend a therapist visit in step 2518 to more directly address theproblems impeding development.

Once the subject is determined to be verbal, as indicated in step 2520,verbal therapies 2522 can be generated by therapy module 2534. Forexample, verbal therapies 2522 can include one or more of languagedrills, articulation exercises, and expressive requesting orcommunicating. Further suggestions of useful devices and products thatmay be helpful for progress may also be provided, and all suggestionscan be tailored to the subject's needs as indicated by the subject'sdiagnosis and progress reports.

As in the non-verbal track, progress in response to verbal therapies iscontinually monitored in step 2524 to determine whether a diagnosis hasimproved at a predicted rate.

If improvement has been measured in step 2524, the system reports on theprogress in step 326 and generates a new recommended therapy 2522 toinduce further improvements.

If no improvement is detected in step 2524, the system can recommendthat the therapy be repeated a predetermined number of times. The systemmay also recommend trying variations in therapy to try and get betterresults. If such repetitions and variations fail, the system canrecommend a therapist visit in step 2528 to more directly address theproblems impeding development.

The steps for non-verbal and verbal therapy can be repeatedindefinitely, to the degree needed to stimulate continued learning andprogress in the subject, and to prevent or retard regress through lossof verbal skills and abilities. While the specific therapy planillustrated in FIG. 25 is directed towards pediatric speech and languagedelay similar plans may be generated for other subjects withdevelopmental or cognitive issues, including plans for adult patients.For example, neurodegenerative conditions and/or age related cognitivedecline may be treated with similar diagnosis and therapy schedules,using treatments selected to be appropriate to such conditions. Furtherconditions that may be treated in adult or pediatric patients by themethods and systems disclosed herein include mood disorders such asdepression, OCD, and schizophrenia; cognitive impairment and decline;sleep disorders; addictive behaviors; eating disorders; and behaviorrelated weight management problems.

FIG. 26 illustrates an overall of data processing flows for a digitalpersonalized medical system comprising a diagnostic module and atherapeutic module, configured to integrate information from multiplesources. Data can include passive data sources (2601), passive data canbe configured to provide more fine grained information, and can comprisedata sets taken over longer periods of time under more naturalconditions. Passive data sources can include for example, data collectedfrom wearable devices, data collected from video feeds (e.g. avideo-enabled toy, a mobile device, eye tracking data from videoplayback), information on the dexterity of a subject based oninformation gathered from three-axis sensors or gyroscopes (e.g. sensorsembedded in toys or other devices that the patient may interact with forexample at home, or under normal conditions outside of a medicalsetting), smart devices that measure any single or combination of thefollowing: subject's speech patterns, motions, touch response time,prosody, lexical vocabulary, facial expressions, and othercharacteristic expressed by the subject. Passive data can comprise dataon the motion or motions of the user, and can include subtle informationthat may or may not be readily detectable to an untrained individual. Insome instances, passive data can provide information that can be moreencompassing.

Passively collected data can comprise data collected continuously from avariety of environments. Passively collected data can provide a morecomplete picture of the subject and thus can improve the quality of anassessment. In some instances, for example, passively collected data caninclude data collected both inside and outside of a medical setting.Passively collected data taken in a medical setting can differ frompassively collected data taken from outside a medical setting.Therefore, continuously collected passive data can comprise a morecomplete picture of a subject's general behavior and mannerisms, andthus can include data or information that a medical practitioner wouldnot otherwise have access to. For example, a subject undergoingevaluation in a medical setting may display symptoms, gestures, orfeatures that are representative of the subject's response to themedical environment, and thus may not provide a complete and accuratepicture of the subject's behavior outside of the medical environmentunder more familiar conditions. The relative importance of one or morefeatures (e.g. features assessed by a diagnostic module) derived from anassessment in the medical environment, may differ from the relativeimportance of one or more features derived from or assessed outside theclinical setting.

Data can comprise information collected through diagnostic tests,diagnostic questions, or questionnaires (2605). In some instances, datafrom diagnostic tests (2605) can comprise data collected from asecondary observer (e.g. a parent, guardian, or individual that is notthe subject being analyzed). Data can include active data sources(2610), for example data collected from devices configured for trackingeye movement, or measuring or analyzing speech patterns.

As illustrated in FIG. 26, data inputs can be fed into a diagnosticmodule which can comprise data analysis (2615) using for example aclassifier, algorithm (e.g. machine learning algorithm), or statisticalmodel, to make a diagnosis of whether the subject is likely to have atested disorder (e.g. Autism Spectrum Disorder) (2620) or is unlikely tohave the tested disorder (2625). The methods and apparatus disclosedherein can alternatively be employed to include a third inconclusivecategory (not depicted in this diagram), which corresponds to thesubject requiring additional evaluation to determine whether he/she isor is not likely to have a tested disorder. The methods and apparatusdisclosed herein are not limited to disorders, and may be applied toother cognitive functions, such as behavioral, neurological, mentalhealth, or developmental conditions. The methods and apparatus mayinitially categorize a subject into one of the three categories, andsubsequently continue with the evaluation of a subject initiallycategorized as “inconclusive” by collecting additional information fromthe subject. Such continued evaluation of a subject initiallycategorized as “inconclusive” may be performed continuously with asingle screening procedure (e.g., containing various assessmentmodules). Alternatively or additionally, a subject identified asbelonging to the inconclusive group may be evaluated using separate,additional screening procedures and/or referred to a clinician forfurther evaluation.

In instances where the subject is determined by the diagnostic model aslikely to have the disorder (2620), a secondary party (e.g. medicalpractitioner, parent, guardian or other individual) may be presentedwith an informative display. An informative display can provide symptomsof the disorder that can be displayed as a graph depicting covariance ofsymptoms displayed by the subject and symptoms displayed by the averagepopulation. A list of characteristics associated with a particulardiagnosis can be displayed with confidence values, correlationcoefficients, or other means for displaying the relationship between asubject's performance and the average population or a populationcomprised of those with a similar disorders.

If the digital personalized medicine system predicts that the user islikely to have a diagnosable condition (e.g. Autism Spectrum Disorder),then a therapy module can provide a behavioral treatment (2630) whichcan comprise behavioral interventions; prescribed activities ortrainings; interventions with medical devices or other therapeutics forspecific durations or, at specific times or instances. As the subjectundergoes the therapy, data (e.g. passive data and diagnostic questiondata) can continue to be collected to perform follow-up assessments, todetermine for example, whether the therapy is working. Collected datacan undergo data analysis (2640) (e.g. analysis using machine learning,statistical modeling, classification tasks, predictive algorithms) tomake determinations about the suitability of a given subject. A growthcurve display can be used to show the subject's progress against abaseline (e.g. against an age-matched cohort). Performance or progressof the individual may be measured to track compliance for the subjectwith a suggested behavioral therapy predicted by the therapy module maybe presented as a historic and predicted performance on a growth curve.Procedures for assessing the performance of an individual subject may berepeated or iterated (2635) until an appropriate behavioral treatment isidentified.

The digital therapeutics treatment methods and apparatus described withreference to FIGS. 23A-23C and FIGS. 24-26 are particularly well suitedfor combination with the methods and apparatus to evaluate subjects withfewer questions described herein with reference to FIGS. 1A to 10. Forexample, the components of diagnosis module 2332 as described herein canbe configured to assess the subject with the decreased set of questionscomprising the most relevant question as described herein, andsubsequently evaluated with the therapy module 2334 to subsequentlyassess the subject with subsequent set of questions comprising the mostrelevant questions for monitoring treatment as described herein.

FIG. 27 shows a system 2700 for evaluating a subject for multipleclinical indications. The system 2700 may comprise a plurality ofcascaded diagnostic modules (such as diagnostic modules 2720, 2730,2740, 2750, and 2760). The cascaded diagnostic modules may beoperatively coupled (such as in a chain of modules) such that an outputfrom one diagnostic module may form an input to another diagnosticmodule. As shown in FIG. 27, the system may comprise a social orbehavioral delay module 2720, an autism or ADHD module 2730, an autismand ADHD discrimination module 2740, a speech or language delay module2750, and an intellectual disability module 2760. Modules (e.g., such asthe diagnostic modules described with respect to FIG. 27) as describedanywhere herein may refer to modules comprising a classifier.Accordingly, a social or behavioral delay module may comprise a socialor behavioral delay classifier, an autism or ADHD module may comprise anautism or ADHD classifier, an autism and ADHD discrimination module maycomprise an autism and ADHD classifier, a speech or language delaymodule may comprise a speech or language delay classifier, anintellectual disability module may comprise an intellectual disabilityclassifier, and so forth.

The social or behavioral delay module 2720 may receive information 2710,such as information from an interactive questionnaire described herein.The social or behavioral delay module may utilize any diagnosticoperations described herein to determine a social or behavioral delaydiagnostic status of the subject. For instance, the social or behavioraldelay module may utilize any operations of the procedure 1300 describedwith respect to FIG. 13 to determine a social or behavioral delaydiagnostic status (i.e., whether or not the subject displays behaviorsconsistent with social or behavioral delay). Upon a determination of thesocial or behavioral delay diagnostic status, the social or behavioraldelay module may output a determination as to whether or not the subjectdisplays social or behavioral delay. The social or behavioral delaymodule may output a positive identification 2722 indicating that thesubject does display social or behavioral delay. The social orbehavioral delay module may output a negative indication 2724 indicatingthat the subject does not display social or behavioral delay. The socialor behavioral delay module may output an inconclusive indication 2726indicating that the social or behavioral delay module has been unable todetermine whether or not the subject displays social or behavioraldelay.

When the social or behavioral delay module determines that the subjectdoes not display social or behavioral delay or that the result of thesocial or behavioral delay inquiry is indeterminate, the system mayoutput such a result and halt its inquiry into the subject's social orbehavioral health.

However, when the social or behavioral delay module determines that thesubject does display social or behavioral delay, the social orbehavioral delay module may pass this result, and information 2710, tothe autism or ADHD module 2730.

The autism or ADHD delay module may utilize any diagnostic operationsdescribed herein to determine an autism or ADHD status of the subject.For instance, the autism or ADHD delay module may utilize any operationsof the procedure 1300 described with respect to FIG. 13 to determine anautism or ADHD diagnostic status (i.e., whether or not the subjectdisplays behaviors consistent with autism or ADHD). Upon a determinationof the autism or ADHD diagnostic status, the autism or ADHD module mayoutput a determination as to whether or not the subject displays autismor ADHD. The autism or ADHD module may output a positive identification2732 indicating that the subject does display autism or ADHD. The autismor ADHD module may output a negative indication 2734 indicating that thesubject does not display autism or ADHD. The autism or ADHD module mayoutput an inconclusive indication 2736 indicating that the autism orADHD module has been unable to determine whether or not the subjectdisplays autism or ADHD.

When the autism or ADHD module determines that the subject does notdisplay autism or ADHD or that the result of the autism or ADHD inquiryis indeterminate, the system may output such a result and halt itsinquiry into the subject's social or behavioral health. In such ascenario, the system may revert to the earlier diagnosis that thesubject displays social or behavioral delay.

However, when the autism or ADHD module determines that the subject doesdisplay autism or ADHD, the autism or ADHD module may pass this result,and information 2710, to the autism and ADHD discrimination module 2740.

The autism and ADHD discrimination module may utilize any diagnosticoperations described herein to discriminate between autism and ADHD. Forinstance, the autism and ADHD discrimination module may utilize anyoperations of the procedure 1300 described with respect to FIG. 13 todiscriminate between autism and ADHD for the subject (i.e., to determinewhether the subject displays behaviors that are more consistent withautism or with ADHD). Upon a discriminating between autism and ADHD, theautism and ADHD discrimination module may output a determination as towhether displays autism or whether the subject displays ADHD. The autismand ADHD discrimination module may output an indication 2742 indicatingthat the subject displays autism. The autism and ADHD discriminationmodule may output an indication 2744 indicating that the subjectdisplays ADHD. The autism and ADHD discrimination module may output aninconclusive indication 2746 indicating that the autism and ADHDdiscrimination module has been unable to discriminate between whetherthe subject's behavior is more consistent with autism or with ADHD.

When the autism and ADHD discrimination module determines that theresult of the autism and ADHD discrimination inquiry is indeterminate,the system may output such a result and halt its inquiry into thesubject's social or behavioral health. In such a scenario, the systemmay revert to the earlier diagnosis that the subject displays behaviorconsistent with autism or ADHD.

Alternatively or in combination, the autism and ADHD discriminationmodule may be further configured to pass information 2710 to one or moreadditional modules. For instance, the autism and ADHD discriminationmodule may be configured to pass information to an obsessive compulsivedisorder module (not shown in FIG. 27). The obsessive compulsivedisorder module may make a determination as to whether a subjectdisplays behavior consistent with obsessive compulsive disorder usingany of the systems and methods described herein (such as any operationsof the procedure 1300).

Alternatively or in combination, the speech or language delay module2750 may receive the information 2710. The speech or language delaymodule may utilize any diagnostic operations described herein todetermine a speech or language delay diagnostic status of the subject.For instance, the speech or language delay module may utilize anyoperations of the procedure 1300 described with respect to FIG. 13 todetermine a speech or language delay diagnostic status (i.e., whether ornot the subject displays behaviors consisting with speech or languagedelay). Upon a determination of the speech or language delay diagnosticstatus, the speech or language delay module may output a determinationas to whether or not the subject displays speech or language delay. Thespeech or language delay module may output a positive identification2752 indicating that the subject does display speech or language delay.The speech or language delay module may output a negative indication2754 indicating that the subject does not display speech or languagedelay. The speech or language delay module may output an inconclusiveindication 2756 indicating that the speech or language delay module hasbeen unable to determine whether or not the subject displays speech orlanguage delay.

When the speech or language delay module determines that the subjectdoes not display speech or language delay or that the result of thespeech or language delay inquiry is indeterminate, the system may outputsuch a result and halt its inquiry into the subject's speech or languagehealth.

However, when the speech or language delay module determines that thesubject does display speech or language delay, the speech or languagedelay module may pass this result, and information 2710, to theintellectual disability module 2760.

The intellectual disability module may utilize any diagnostic operationsdescribed herein to determine an intellectual disability status of thesubject. For instance, the intellectual disability module may utilizeany operations of the procedure 1300 described with respect to FIG. 13to determine an intellectual disability diagnostic status (i.e., whetheror not the subject displays behaviors consistent with intellectualdisability). Upon a determination of the intellectual disabilitydiagnostic status, the intellectual disability module may output adetermination as to whether or not the subject displays intellectualdisability. The intellectual disability module may output a positiveidentification 2762 indicating that the subject does displayintellectual disability. The intellectual disability module may output anegative indication 2764 indicating that the subject does not displayintellectual disability. The intellectual disability module may outputan inconclusive indication 2766 indicating that the intellectualdisability module has been unable to determine whether or not thesubject displays intellectual disability.

When the intellectual disability module determines that the subject doesnot display intellectual disability or that the result of theintellectual disability inquiry is indeterminate, the system may outputsuch a result and halt its inquiry into the subject's speech or languagehealth. In such a scenario, the system may revert to the earlierdiagnosis that the subject displays speech or language delay.

Alternatively or in combination, the intellectual disability module maybe further configured to pass information 2710 to one or more additionalmodules. For instance, the intellectual disability module may beconfigured to pass information to a dyslexia module (not shown in FIG.27). The dyslexia module may make a determination as to whether asubject displays behavior consistent with dyslexia using any of thesystems and methods described herein (such as any operations of theprocedure 1300).

Though described with reference to social or behavioral delay, autism,ADHD, obsessive compulsive disorder, speech or language delay,intellectual disability, and dyslexia, the system 2700 may comprise anynumber of modules (such as 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or more than10 modules) that may provide a diagnostic status for any behavioraldisorder. The modules may be operatively coupled (such as cascaded orchained) in any possible order.

The systems and methods described anywhere herein may be used as a basisfor a treatment plan, or for administration of a drug, for a disorderdiagnosed by any system or method for diagnosis described herein.

The systems and methods described anywhere herein may be used toadminister a drug to treat acute stress disorder, such as propranolol,citalopram, escitalopram, sertraline, paroxetine, fluoextine,venlafaxine, mirtazapine, nefazodone, carbamazepine, divalproex,lamotrigine, topiramate, prazosin, phenelzine, imipramine, diazepam,clonazepam, lorazepam, or alprazolam.

The systems and methods described anywhere herein may be used toadminister a drug to treat adjustment disorder, such as busiprone,escitalopram, sertraline, paroxetine, fluoextine, diazepam, clonazepam,lorazepam, or alprazolam.

The systems and methods described anywhere herein may be used toadminister a drug to treat agoraphobia, such as diazepam, clonazepam,lorazepam, alprazolam, citalopram, escitalopram, sertraline, paroxetine,fluoextine, or busiprone.

The systems and methods described anywhere herein may be used toadminister a drug to treat Alzheimer's disease, such as donepezil,galantamine, memantine, or rivastigmine.

The systems and methods described anywhere herein may be used toadminister a drug to treat anorexia nervosa, such as olanzapine,citalopram, escitalopram, sertraline, paroxetine, or fluoextine.

The systems and methods described anywhere herein may be used toadminister a drug to treat anxiety disorders, such as sertraline,escitalopram, citalopram, fluoxetine, diazepam, buspirone, venlafaxine,duloxetine, imipramine, desipramine, clomipramine, lorazepam,clonazepam, or pregabalin.

The systems and methods described anywhere herein may be used toadminister a drug to treat attachment disorder.

The systems and methods described anywhere herein may be used toadminister a drug to treat attention deficit/hyperactivitydisorder(ADHD/ADD), such as amphetamine (for instance, in a dosage of 5mg to 50 mg), dextroamphetamine (for instance, in a dosage of 5 mg to 60mg), methylphenidate (for instance, in a dosage of 5 mg to 60 mg),methamphetamine (for instance, in a dosage of 5 mg to 25 mg),dexmethylphenidate (for instance, in a dosage of 2.5 mg to 40 mg),guanfacine (for instance, in a dosage of 1 mg to 10 mg), atomoxetine(for instance, in a dosage of 10 mg to 100 mg), lisdexamfetamine (forinstance, in a dosage of 30 mg to 70 mg), clonidine (for instance, in adosage of 0.1 mg to 0.5 mg), or modafinil (for instance, in a dosage of100 mg to 500 mg).

The systems and methods described anywhere herein may be used toadminister a drug to treat autism or autism spectrum disorders, such asrisperidone (for instance, in a dosage of 0.5 mg to 20 mg), quetiapine(for instance, in a dosage of 25 mg to 1000 mg), amphetamine (forinstance, in a dosage of 5 mg to 50 mg), dextroamphetamine (forinstance, in a dosage of 5 mg to 60 mg), methylphenidate (for instance,in a dosage of 5 mg to 60 mg), methamphetamine (for instance, in adosage of 5 mg to 25 mg), dexmethylphenidate (for instance, in a dosageof 2.5 mg to 40 mg), guanfacine (for instance, in a dosage of 1 mg to 10mg), atomoxetine (for instance, in a dosage of 10 mg to 100 mg),lisdexamfetamine (for instance, in a dosage of 30 mg to 70 mg),clonidine (for instance, in a dosage of 0.1 mg to 0.5 mg), oraripiprazole (for instance, in a dosage of 1 mg to 10 mg).

The systems and methods described anywhere herein may be used toadminister a drug to treat bereavement, such as citalopram, duloxetine,or doxepin.

The systems and methods described anywhere herein may be used toadminister a drug to treat binge eating disorder, such aslisdexamfetamine.

The systems and methods described anywhere herein may be used toadminister a drug to treat bipolar disorder, such as topiramate,lamotrigine, oxcarbazepine, haloperidol, risperidone, quetiapine,olanzapine, aripiprazole, or fluoxetine.

The systems and methods described anywhere herein may be used toadminister a drug to treat body dysmorphic disorder, such as sertraline,escitalopram, or citalopram.

The systems and methods described anywhere herein may be used toadminister a drug to treat brief psychotic disorder, such as clozapine,asenapine, olanzapine, or quetiapine.

The systems and methods described anywhere herein may be used toadminister a drug to treat bulimia nervosa, such as sertraline, orfluoxetine.

The systems and methods described anywhere herein may be used toadminister a drug to treat conduct disorder, such as lorazepam,diazepam, or clobazam.

The systems and methods described anywhere herein may be used toadminister a drug to treat cyclothymic disorder.

The systems and methods described anywhere herein may be used toadminister a drug to treat delusional disorder, such as clozapine,asenapine, risperidone, venlafaxine, bupropion, or buspirone.

The systems and methods described anywhere herein may be used toadminister a drug to treat depersonalization disorder, such assertraline, fluoxetine, alprazolam, diazepam, or citalopram.

The systems and methods described anywhere herein may be used toadminister a drug to treat depression, such as sertraline, fluoxetine,citalopram, bupropion, escitalopram, venlafaxine, aripiprazole,buspirone, vortioxetine, or vilazodone.

The systems and methods described anywhere herein may be used toadminister a drug to treat disinhibited social engagement disorder.

The systems and methods described anywhere herein may be used toadminister a drug to treat disruptive mood dysregulation disorder, suchas quetiapine, clozapine, asenapine, or pimavanserin.

The systems and methods described anywhere herein may be used toadminister a drug to treat dissociative amnesia, such as alprazolam,diazepam, lorazepam, or chlordiazepoxide.

The systems and methods described anywhere herein may be used toadminister a drug to treat dissociative disorder, such as bupropion,vortioxetine, or vilazodone.

The systems and methods described anywhere herein may be used toadminister a drug to treat dissociative fugue, such as amobarbital,aprobarbital, butabarbital, or methohexitlal.

The systems and methods described anywhere herein may be used toadminister a drug to treat dissociative identity disorder.

The systems and methods described anywhere herein may be used toadminister a drug to treat dyslexia, such as amphetamine (for instance,in a dosage of 5 mg to 50 mg), dextroamphetamine (for instance, in adosage of 5 mg to 60 mg), methylphenidate (for instance, in a dosage of5 mg to 60 mg), methamphetamine (for instance, in a dosage of 5 mg to 25mg), dexmethylphenidate (for instance, in a dosage of 2.5 mg to 40 mg),guanfacine (for instance, in a dosage of 1 mg to 10 mg), atomoxetine(for instance, in a dosage of 10 mg to 100 mg), lisdexamfetamine (forinstance, in a dosage of 30 mg to 70 mg), clonidine (for instance, in adosage of 0.1 mg to 0.5 mg), or modafinil (for instance, in a dosage of100 mg to 500 mg).

The systems and methods described anywhere herein may be used toadminister a drug to treat dysthymic disorder, such as bupropion,venlafaxine, sertraline, or citalopram.

The systems and methods described anywhere herein may be used toadminister a drug to treat eating disorders, such as olanzapine,citalopram, escitalopram, sertraline, paroxetine, or fluoextine.

The systems and methods described anywhere herein may be used toadminister a drug to treat expressive language disorder.

The systems and methods described anywhere herein may be used toadminister a drug to treat gender dysphoria, such as estrogen,prostogen, or testosterone.

The systems and methods described anywhere herein may be used toadminister a drug to treat generalized anxiety disorder, such asvenlafaxine, duloxetine, buspirone, sertraline, or fluoxetine.

The systems and methods described anywhere herein may be used toadminister a drug to treat hoarding disorder, such as buspirone,sertraline, escitalopram, citalopram, fluoxetine, paroxetine,venlafaxine, or clomipramine.

The systems and methods described anywhere herein may be used toadminister a drug to treat intellectual disability.

The systems and methods described anywhere herein may be used toadminister a drug to treat intermittent explosive disorder, such asasenapine, clozapine, olanzapine, or pimavanserin.

The systems and methods described anywhere herein may be used toadminister a drug to treat kleptomania, such as escitalopram,fluvoxamine, fluoxetine, or paroxetine.

The systems and methods described anywhere herein may be used toadminister a drug to treat mathematics disorder.

The systems and methods described anywhere herein may be used toadminister a drug to treat obsessive-compulsive disorder, such asbuspirone (for instance, in a dosage of 5 mg to 60 mg), sertraline (forinstance, in a dosage of up to 200 mg), escitalopram (for instance, in adosage of up to 40 mg), citalopram (for instance, in a dosage of up to40 mg), fluoxetine (for instance, in a dosage of 40 mg to 80 mg),paroxetine (for instance, in a dosage of 40 mg to 60 mg), venlafaxine(for instance, in a dosage of up to 375 mg), clomipramine (for instance,in a dosage of up to 250 mg), or fluvoxamine (for instance, in a dosageof up to 300 mg).

The systems and methods described anywhere herein may be used toadminister a drug to treat oppositional defiant disorder.

The systems and methods described anywhere herein may be used toadminister a drug to treat panic disorder, such as bupropion,vilazodone, or vortioxetine.

The systems and methods described anywhere herein may be used toadminister a drug to treat Parkinson's disease, such as rivastigmine,selegiline, rasagiline, bromocriptine, amantadine, cabergoline, orbenztropine.

The systems and methods described anywhere herein may be used toadminister a drug to treat pathological gambling, such as bupropion,vilazodone, or vartioxetine.

The systems and methods described anywhere herein may be used toadminister a drug to treat pica.

The systems and methods described anywhere herein may be used toadminister a drug to treat postpartum depression, such as sertraline,fluoxetine, citalopram, bupropion, escitalopram, venlafaxine,aripiprazole, buspirone, vortioxetine, or vilazodone.

The systems and methods described anywhere herein may be used toadminister a drug to treat posttraumatic stress disorder, such assertraline, fluoxetine, or paroxetine.

The systems and methods described anywhere herein may be used toadminister a drug to treat premenstrual dysphoric disorder, such asestadiol, drospirenone, sertraline, citalopram, fluoxetine, orbusiprone.

The systems and methods described anywhere herein may be used toadminister a drug to treat pseudobulbar affect, such as dextromethorphanhydrobromide, or quinidine sulfate.

The systems and methods described anywhere herein may be used toadminister a drug to treat pyromania, such as clozapine, asenapine,olanzapine, paliperidone, or quetiapine.

The systems and methods described anywhere herein may be used toadminister a drug to treat reactive attachment disorder.

The systems and methods described anywhere herein may be used toadminister a drug to treat reading disorder.

The systems and methods described anywhere herein may be used toadminister a drug to treat rett's syndrome.

The systems and methods described anywhere herein may be used toadminister a drug to treat rumination disorder.

The systems and methods described anywhere herein may be used toadminister a drug to treat schizoaffective disorder, such as sertraline,carbamazepine, oxcarbazepine, valproate, haloperidol, olanzapine, orloxapine.

The systems and methods described anywhere herein may be used toadminister a drug to treat schizophrenia, such as chlopromazine,haloperidol, fluphenazine, risperidone, quetiapine, ziprasidone,olanzapine, perphenazine, aripiprazole, or prochlorperazine.

The systems and methods described anywhere herein may be used toadminister a drug to treat schizophreniform disorder, such aspaliperidone, clozapine, risperidone.

The systems and methods described anywhere herein may be used toadminister a drug to treat seasonal affective disorder, such assertraline, or fluoxetine.

The systems and methods described anywhere herein may be used toadminister a drug to treat separation anxiety disorder.

The systems and methods described anywhere herein may be used toadminister a drug to treat shared psychotic disorder, such as clozapine,pimavanserin, risperidone, or lurasidone.

The systems and methods described anywhere herein may be used toadminister a drug to treat social (pragmatic) communication disorder.

The systems and methods described anywhere herein may be used toadminister a drug to treat social anxiety phobia, such as amitriptyline,bupropion, citalopram, fluoxetine, sertraline, or venlafaxine.

The systems and methods described anywhere herein may be used toadminister a drug to treat somatic symptom disorder.

The systems and methods described anywhere herein may be used toadminister a drug to treat specific phobia, such as diazepam, estazolam,quazepam, or alprazolam.

The systems and methods described anywhere herein may be used toadminister a drug to treat stereotypic movement disorder, such asrisperidone, or clozapine.

The systems and methods described anywhere herein may be used toadminister a drug to treat stuttering.

The systems and methods described anywhere herein may be used toadminister a drug to treat Tourette's disorder, such as haloperidol,fluphenazine, risperidone, ziprasidone, pimozide, perphenazine, oraripiprazole.

The systems and methods described anywhere herein may be used toadminister a drug to treat transient tic disorder, such as guanfacine,clonidine, pimozide, risperidone, citalopram, escitalopram, sertraline,paroxetine, or fluoextine.

FIG. 28 shows a drug that may be administered in response to a diagnosisby the systems and methods described herein. The drug may be containedwithin a container 2800, such as a pill bottle. The container may have alabel 2810 bearing instructions “If diagnosed with disorder x,administer drug y”. The disorder x may be any disorder described herein.The drug y may be any drug described herein.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

What is claimed is:
 1. A computer-implemented method for determining atreatment for an individual for a neurological disorder, said methodcomprising: (a) receiving data of said individual related to saidneurological disorder; and (b) evaluating said data using at least oneclassifier to select at least one therapeutic agent for treating saidneurological disorder.
 2. The method of claim 1, wherein said at leastone therapeutic agent comprises a stimulant or antipsychotic drug fortreating said neurological disorder.
 3. The method of claim 1, whereinsaid at least one therapeutic agent comprises amphetamine or anamphetamine-derived drug.
 4. The method of claim 1, wherein said atleast one therapeutic agent is selected from the group consisting ofrisperidone, quetiapine, amphetamine, dextroamphetamine,methylphenidate, methamphetamine, dextroamphetamine, dexmethylphenidate,guanfacine, atomoxetine, lisdexamfetamine, clonidine, and aripiprazole,and modafinil.
 5. The method of claim 1, wherein said neurologicaldisorder is selected from the group consisting of autism spectrumdisorder, attention deficit disorder, attention deficit hyperactivitydisorder, and dyslexia.
 6. The method of claim 1, wherein said datacomprises active data generated from at least one active data source andpassive data generated from at least one passive data source.
 7. Themethod of claim 6, wherein said passive data comprises passive datastreams from user interactions with at least one of an activity, game,mobile device or application, smart toy, wearable sensor, and activitymonitor.
 8. The method of claim 1, wherein said data comprisesinformation acquired from at least one of genetic data, floral data, asleep monitor, and eye tracking of said individual.
 9. The method ofclaim 1, wherein said data comprises at least one of demographic dataand answers to a set of diagnostic questions.
 10. The method of claim 1,wherein said at least one classifier comprises an assessment model forproviding an evaluation result based on said data, wherein saidevaluation result is a first categorical determination or a firstinconclusive determination with respect to a presence or absence of saidneurological disorder.
 11. The method of claim 10, wherein said firstcategorical determination for said presence or absence of saidneurological disorder in said individual is based on a specifiedsensitivity and a specified specificity.
 12. The method of claim 10,wherein said at least one classifier comprises a subset of a pluralityof tunable machine learning models.
 13. The method of claim 12, furthercomprising: (a) requesting additional data when said evaluation resultcomprises said first inconclusive determination; and (b) generating asecond categorical determination or a second inconclusive determinationbased on said additional data using at least one additional machinelearning model selected from said plurality of tunable machine learningmodels.
 14. The method of claim 12, further comprising: (a) combiningscores for each of said subset of said plurality of tunable machinelearning models to generate a combined preliminary output score; and (b)mapping said combined preliminary output score to said first categoricaldetermination or to said first inconclusive determination for saidpresence or absence of said neurological disorder in said individual.15. The method of claim 1, wherein said at least one classifiercomprises a chain of classifiers for providing an evaluation result forsaid neurological disorder based on said data.
 16. The method of claim15, wherein said chain of classifiers comprises a first classifier thatgenerates a first output based on said data and a second classifier thatgenerates a second output based on said first output.
 17. The method ofclaim 1, wherein said at least one classifier comprises a therapeuticmodel for selecting said at least one therapeutic agent.
 18. The methodof claim 17, wherein said at least one classifier generates a personaltherapeutic treatment plan for said individual.
 19. The method of claim18, further comprising receiving feedback data based on performance ofsaid personal therapeutic treatment plan and updating said personaltherapeutic treatment plan based on said feedback data.
 20. The methodof claim 18, wherein said personal therapeutic treatment plan comprisesa drug therapy and digital therapeutics.